Tap Execution & Isolation

A tap’s fetch() is Python that Datris executes on your server. By default it runs in-process inside the server. For stronger separation, Datris can instead run each tap in a dedicated, isolated runner container, cut off from the platform’s credentials and internal services.

On docker compose / prod, the isolated runner is on by default (USE_TAP_RUNNER=true). sbt / IDE runs (no sidecar) stay in-process. Tap behavior is identical either way — the same fetch() contract, the same environment variables, the same packages. The only difference is where the code runs and what it can reach. scripts/install.sh and docker/vault-init.sh mint TAP_RUNNER_TOKEN; the server refuses to start with isolation on and a missing or changeme token.

Execution modes

	In-process (`sbt` / IDE, or `USE_TAP_RUNNER=false`)	Isolated runner (compose/prod default)
Where tap code runs	Inside the datris server container	A separate `datris-tap-runner` container
Platform secrets reachable	Limited (see always-on protections)	No
Internal services (database, object store, secrets store) reachable	Yes	No
Internet + platform API reachable	Yes	Yes
Extra packages	Installed in an isolated, throwaway environment	Installed in an isolated, throwaway environment

In-process mode runs tap code inside the server. Tap code in this mode shares the server’s filesystem and network, so it can reach internal services. This is the sbt / IDE path (no sidecar). Compose/prod should keep the isolated runner on. In isolated runner mode (USE_TAP_RUNNER=true), tap code runs in a container that holds no platform credentials and sits on a network with no route to the platform’s internal services. Even a misbehaving tap has nothing to read and nowhere to pivot — it can fetch from its source, install the packages it declares, call back into the Datris API, and return records. That is all. Enable it when you want defense-in-depth beyond the always-on protections.

The runner container

The isolated runner is a dedicated datris-tap-runner sidecar — a small, standalone container built from its own minimal Python image (not the server image). It is hardened to hold and grant as little as possible:

No platform secrets. The image carries no secrets-store credentials, no .env file, and no AI/database credentials — there is nothing sensitive inside it to read.
Network-isolated. It runs on a dedicated network with no route to the platform’s internal services (database, object store, secrets store). It can reach the public internet and the Datris API — nothing else internal.
Non-root, read-only filesystem. Tap code runs as an unprivileged user on a read-only root filesystem, with all Linux capabilities dropped and privilege escalation disabled.
Ephemeral scratch. The only writable area is a small in-memory scratch space that is wiped after every run, so nothing a tap writes — temp files, installed packages — survives into the next run.

This is operating-system container isolation — separate process, filesystem, and network namespaces sharing the host kernel. It is the right boundary for a self-hosted, single-operator deployment. It is not a virtual-machine boundary: a kernel-level container escape would cross it, so running fully untrusted, multi-tenant tap code calls for stronger per-run isolation — a user-space kernel such as gVisor (runsc), or a microVM (Firecracker / Kata). Datris does not ship or configure these. But because the runner is a standard OCI container, you can run it under a hardened runtime yourself if your host provides one — for example, with gVisor installed, run the datris-tap-runner container under runsc (Docker’s --runtime=runsc, or runtime: runsc on the service). Datris hasn’t validated this path, so treat it as an advanced, self-managed hardening step.

Always-on protections

These apply in both modes, regardless of the runner setting:

Platform secrets are never placed in a tap’s environment. The credentials the platform uses for itself (secrets-store token, AI provider keys, database passwords) are stripped from the tap’s environment. A tap receives only Datris-injected variables (DATRIS_*), any per-run parameters, and its own configured secret — nothing else.
Extra packages install in isolation. Packages a tap declares are installed into a throwaway virtual environment, never into the system Python. The pre-installed common packages remain available; the per-tap environment is discarded after the run.
Private-network egress is blocked. HTTP taps and REST endpoint destinations refuse to connect to loopback, link-local, private, or cloud-metadata addresses, so a tap cannot be pointed at internal services. If your endpoints legitimately live on an internal network, set DATRIS_ALLOW_PRIVATE_EGRESS=true in .env (see the Configuration Reference).

Configuration

Set these on the datris service (for example in your .env). Defaults work out of the box.

Variable	Default	Purpose
`USE_TAP_RUNNER`	`true` (compose) / unset=`false` (sbt)	Isolated runner. Set `false` to run in-process.
`TAP_RUNNER_URL`	`http://datris-tap-runner:8090`	Address of the runner service.
`TAP_RUNNER_TOKEN`	minted on first boot	Shared token authenticating the server to runner call. install.sh / vault-init mint it; changeme or empty fails fast when isolation is on.
`TAP_RUNNER_CALLBACK_HOST`	`datris`	Host a tap uses to call back into the platform API from the runner. The datris service name on the shared network.

The runner is defined as the datris-tap-runner service in docker-compose.yml; a normal docker compose up starts it alongside the server and routes taps there unless you set USE_TAP_RUNNER=false. To force in-process (for example a no-sidecar sbt run):

USE_TAP_RUNNER=false sbt -Dconfig.file=... run

What changes for tap authors

Almost nothing. Taps fetch data and return records; the platform writes them to the destination pipeline. Reading a source over the internet, installing declared packages, using the tap’s own secret, and calling the Datris API via DATRIS_PLATFORM_HOST / DATRIS_PLATFORM_PORT all work the same in both modes. The one thing to know: in isolated runner mode a tap cannot open a direct connection to the platform’s internal database / object store by hostname — those live on a network the runner can’t reach. This is intentional, and it’s not how taps are meant to work anyway: a tap returns records and lets the pipeline handle the destination. If you have a legacy tap that connected directly to an internal service, run it in-process or rework it to return records.

Reading platform data from a tap

A tap can read data already in Datris — for example a stock_tickers table — to drive its fetch. Do this through the query API, using the injected platform host and database name. The same code works in both execution modes, because DATRIS_PLATFORM_HOST is set correctly for each:

import os, json, urllib.request

def fetch():
    host = os.environ["DATRIS_PLATFORM_HOST"]   # "localhost" in-process, "datris" in the runner
    port = os.environ["DATRIS_PLATFORM_PORT"]
    db   = os.environ["DATRIS_POSTGRES_DATABASE"]
    req = urllib.request.Request(
        f"http://{host}:{port}/api/v1/query/postgres",
        data=json.dumps({"sql": "SELECT ticker FROM stock_tickers", "database": db}).encode(),
        headers={"Content-Type": "application/json"},
    )
    tickers = json.loads(urllib.request.urlopen(req).read())["results"]
    # ...use tickers to drive the rest of fetch()...
    return tickers

Note there is no credential in that call. When the deployment requires API keys, the platform mints a per-run token (DATRIS_PLATFORM_TOKEN) and the tap wrapper attaches it to requests / urllib calls aimed at DATRIS_PLATFORM_HOST automatically — the script above works unchanged in both modes and with keys on or off. The token is read-only and scoped to the run; the audit log records the callback as the tap. Use /api/v1/query/mongodb (or /api/v1/query/natural) the same way for Mongo-backed data. In the isolated runner, a tap cannot instead open a direct database connection (e.g. to postgres:5432) — internal services aren’t reachable from the runner’s network. Querying through the API is the supported path and works regardless of mode.

When to use which

Isolated runner (compose/prod default) — tap code stays separated from platform credentials and internal services. Isolated taps cannot open a direct connection to internal DB / MinIO / Vault; they return records and the pipeline writes destinations.
In-process (sbt / IDE, or USE_TAP_RUNNER=false) — tap fetch() runs inside the server JVM. Fine for local development without the sidecar, or a legacy tap that still opens a direct internal connection. The server logs a loud warning. Do not treat a laptop compose install as in-process — compose isolates by default.

Getting Started

Connect your agent

Assistant

Catalog

Taps

Ingestion

Destinations

Data Quality

Transformation

AI Features

Configuration

Governance & Security

Production

Integrations

Recipes

Examples

Tap Execution & Isolation

Execution modes

The runner container

Always-on protections

Configuration

What changes for tap authors

Reading platform data from a tap

When to use which

​Execution modes

​The runner container

​Always-on protections

​Configuration

​What changes for tap authors

​Reading platform data from a tap

​When to use which

Execution modes

The runner container

Always-on protections

Configuration

What changes for tap authors

Reading platform data from a tap

When to use which