Incidents & Recovery Agent

With the recovery agent on, Datris stops waiting for you to notice a broken data flow. When a scheduled tap finally fails (after the platform’s own retries), a pipeline job errors, a tap goes stale, a pipeline’s volume swings hard, or a tap’s columns change, the platform opens an incident, diagnoses it with the same judgment as the Ops chat, and proposes a fix. Every action passes through the agent policy. The platform rehearses the fix in a scratch lane, shows the before/after evidence, then lands it with a real run. If verification fails it reverts the fix. Every step is recorded on the incident. The loop is: observe → diagnose → propose → rehearse → gate → execute → verify → record. The AI proposes; deterministic platform code decides. Limits, verification and the revert are code the model cannot talk its way past.

​

Enabling it

Requires the agent policy (USE_AGENT_POLICY=true) for its approval flow.

1. Add to your .env:
   RECOVERY_AGENT_ENABLED=true
   # optional: JSON POST on incident open / awaiting_approval / resolved / failed / abandoned
   # (also receives doctor_error / doctor_recovered when DOCTOR_INTERVAL_MINUTES is set);
   # incident events carry evidenceSummary, one line of numbers, once evidence exists
   INCIDENT_WEBHOOK_URL=
   
2. Recreate the Datris container:
   docker compose up -d --force-recreate datris
   
3. Set the recovery mode in Configuration → Agent Policy → Recovery: choose a mode, a verify mode and whether schema-drift incidents may act, then Save recovery settings. It ships as off, so nothing changes until you choose:

A per-tap or per-pipeline override can put one resource on a different mode — one critical tap on propose while everything else runs autopilot, or the reverse. The Agent Policy page shows the current recovery block next to the one recommended for this install’s environment label: autopilot with loose thresholds on dev, propose with strict thresholds on qa, production and unlabelled installs. An admin can apply the recommendation with one click after a confirmation. Switching to autopilot asks you to confirm. The mode never changes on its own.

​

What opens an incident

Guards, all enforced in code: one open incident per resource; a resource whose latest run is healthy never gets one; a cooldown (default 6 h) after a failed or abandoned attempt; a cap on simultaneous open incidents. Volume and schema-drift incidents are diagnosis-only by default: the value is the triage, not an action (see Schema drift incidents).

​

What the agent may do

During diagnosis it holds read-only tools, so it cannot change anything while investigating. They include the doctor, lineage, provenance, data discovery and pipeline version history, so its summary says what a change touches. For a tap, its proposal may only contain: re-run the tap, fix the tap script (then test it), or “this needs a human” with an explanation. For a pipeline, its only action is regenerating a stored AI script (see Pipeline incidents). Deletes, schema migrations and secret changes are structurally impossible for it. Script fixes land as a new version, so history is never lost. Every call it makes runs as its own recovery-agent API key and is tagged with the incident id, so with the audit log on, the audit log is the incident’s complete ledger (strict mode guarantees no entry is dropped). Revoking that key stops the agent entirely, independent of any flag.

​

What you see before you approve

Incidents with platform-computed evidence show an evidence card on the incident in Activity → Incidents and, compact, on its approval above Approve / Reject. Platform code computes it, not a model: Column values are never shown on the card, in the webhook or to the model when DATRIS_AI_SAMPLE_VALUES=false. Incidents opened before the upgrade have no evidence and render as before. An incident waiting on a person closes itself as resolved (“recovered outside the agent”) when the tap or pipeline next runs healthy, for example after you recreate a rejected secret and re-run the tap, or on its next scheduled run. Only a run that started after the agent’s own last run counts. The closing step says when the run happened and what started it (manual or cron). Any queued actions the incident was waiting on are withdrawn from Activity → Approvals, and the step says how many. If one of those actions was approved or rejected in the meantime, or someone chose to proceed past the card, the incident follows that decision instead. A resolved incident starts no cooldown.

​

Scratch-first verification

With recovery.verifyMode at its default scratch-first, a proposed fix is verified in two stages:

1. Scratch rehearsal. The proposed script runs in test mode: the source is called, the incremental cursor is not committed, and the tap’s last-run status is not touched. Its output goes through the real pipeline with the destination swapped for the scratch lane (the Live Read destination). The landed rows are profiled and compared with the last good run against the thresholds. In propose mode this happens before the approval is queued, so the approval already carries the evidence. A rehearsal that passes (or waits for a person) counts as the proposed script’s test run, so the approved fix can be saved on a scheduled tap without clearing its schedule first. While a fix waits for approval, don’t run a test of that tap yourself: a manual test replaces the rehearsal’s record, and the approved save is then refused. If the fix is rejected or never saved, the tap gets its own last test back.
2. Real run. Only a pass continues. The fix is applied (directly or after approval) and the tap must complete a real run healthily; when it feeds a pipeline the load must finish without error. The real run’s landed data is profiled into the evidence too.

The verdict decides what happens next: If the real run fails verification, the platform reverts the script to the pre-incident version, closes the incident as failed, and starts the cooldown. verifyMode: real skips the rehearsal and goes straight to the real run.

​

Schema drift incidents

A schema-drift incident opens when a scheduled tap run succeeds but returns a different set of columns than the run before it. By default it is diagnosis-only: one AI diagnosis explains the change and the incident resolves as triaged. Set recovery.allowSchemaDriftActions to true to let a schema-drift incident on a tap in autopilot propose a script fix and run it through scratch-first verification. Destination schema changes always stay with a person.

​

Credential and platform failures

Before any AI diagnosis the platform runs the doctor’s quick check itself. An error that stops any run from succeeding (a full disk or staging area, the strict audit log refusing entries, the script sidecar down, runs turned away by the runner queue, a missing primary AI secret) classifies the incident platform and hands it to a person with the doctor’s fix, without spending an AI call. Other doctor errors are passed to the diagnosis as context. A failure where the source rejected the credentials (401, 403, an expired or invalid token), or where the tap’s secret is missing or empty, is classified credential in code. No tap actions are proposed, and the evidence card names the Secret to check (never its value) with a link to Configuration → Secrets. Re-running a tap with a bad key would fail the same way.

​

Pipeline incidents

For a pipeline whose AI data-quality or transformation script fails, the only action is regenerating that stored script from the pipeline’s current instruction (the pipeline:update:codegen-scripts policy action).

• Fed by a tap: the fix is verified through the feeding tap, scratch lane first, then a real run. If verification fails, the script is restored to the version it had when the incident proposed the fix, including over any change a person made while the proposal waited for approval.
• Fed by uploads or another source the platform cannot re-feed: the incident closes as resolved but unverified. The next run shows whether the fix worked.

Destination, schema and upload problems go to a person.

​

Learning from incidents

The diagnosis may include a learnNote, a source quirk worth remembering (for example, that a source returns timestamps without an offset). When an incident resolves after a tap script fix (update_tap), the platform saves the note as a suggested tap prompt fragment named incident-<id>. Its alias is the tap’s source host. The incident’s timeline records “learn note saved as suggested prompt fragment incident-<id>”. A suggested fragment is stored disabled, so it never reaches a prompt until a person accepts it. List suggestions with GET /api/v1/tap-prompts?status=suggested and accept one from a user session with POST /api/v1/tap-prompts and "enabled": true, "status": "active". Agents can list suggestions but cannot accept them. Notes are capped at 2,000 characters and pass through the same scrubber as run errors when DATRIS_AI_SAMPLE_VALUES=false. Resolutions by a re-run only, and failed or abandoned incidents, save nothing.

​

Limits

Set under the agent policy’s recovery section; enforced by the runner, not the prompt. The verification thresholds (maxRecordDeltaPct, allowColumnRemoval and the rest) are in the Recovery mode table.

​

Watching it

• Activity → Incidents: every incident with its state, classification, step-by-step narrative, and any approvals it waits on; a person can abandon one at any time.
• Evidence card: see What you see before you approve.
• Webhook: with INCIDENT_WEBHOOK_URL set, every incident event is a JSON POST with event, incidentId, kind, resourceType, resource, detail and ts. Once the incident has evidence it also carries evidenceSummary: one line of counts (verdict, records before and after, columns added and removed, DQ change, downstream pipelines, failed checks), never a column value.
• Ops chat: incidents the platform is working appear in the assistant’s context — ask “what happened to the prices tap?” and it explains the incident record instead of re-diagnosing.
• Agents: list_incidents, get_incident and get_incident_evidence MCP tools (read-only: only the platform opens incidents).
• Metrics: datris_incidents_total{kind,outcome} and datris_incident_duration_seconds on the Prometheus endpoint.

​

Related

• Agent Policy — the gate every recovery action passes through
• Audit Log — the incident’s ledger, joined on the incident id
• Tap Prompt Fragments — where accepted learn notes go to work