Tap Execution & Isolation

​

Execution modes

​

The runner container

• No platform secrets. The image carries no secrets-store credentials, no .env file, and no AI/database credentials — there is nothing sensitive inside it to read.
• Network-isolated. It runs on a dedicated network with no route to the platform’s internal services (database, object store, secrets store). It can reach the public internet and the Datris API — nothing else internal.
• Non-root, read-only filesystem. Tap code runs as an unprivileged user on a read-only root filesystem, with all Linux capabilities dropped and privilege escalation disabled.
• Ephemeral scratch. The only writable area is a small in-memory scratch space (512 MB by default, a tmpfs mount, so it comes out of RAM rather than disk) that is wiped after every run, so nothing a tap writes — temp files, installed packages — survives into the next run. The per-run venv and any pip-installed packages live there too, so a script must stream the source, never download it: a fetch() that saves a large file to disk (or lets a library spool one under TMPDIR) fails with No space left on device. Read remote files directly over HTTP in chunks or row groups and yield records as they arrive. The 512 MB figure is set on the runner’s tmpfs entry in the compose file; raise it only on hosts with the RAM to spare.

​

Always-on protections

• Platform secrets are never placed in a tap’s environment. The credentials the platform uses for itself (secrets-store token, AI provider keys, database passwords) are stripped from the tap’s environment. A tap receives only Datris-injected variables (DATRIS_*), any per-run parameters, and its own configured secret — nothing else.
• Extra packages install in isolation. Packages a tap declares are installed into a throwaway virtual environment, never into the system Python. The pre-installed common packages remain available; the per-tap environment is discarded after the run.
• Private-network egress is blocked. HTTP taps and REST endpoint destinations refuse to connect to loopback, link-local, private, or cloud-metadata addresses, so a tap cannot be pointed at internal services. If your endpoints legitimately live on an internal network, set DATRIS_ALLOW_PRIVATE_EGRESS=true in .env (see the Configuration Reference).

​

Overload protection

• Bounded concurrency. Each runner runs at most TAP_RUNNER_MAX_CONCURRENT scripts at once (default 4; CODEGEN_RUNNER_MAX_CONCURRENT for datris-codegen-runner). A run that arrives when every slot is taken is answered “busy” at once, and nothing is started for it.
• The server waits its turn. A busy answer is retried with a short backoff (never more than 30 seconds between attempts) for up to TAP_RUNNER_QUEUE_TIMEOUT_SECONDS (default 600). A tap run that waited shows the time in its run history, for example queued 12.3s next to the duration (the duration includes the wait). A run that still has no slot after the timeout fails with one sentence naming TAP_RUNNER_MAX_CONCURRENT: raise it, or stagger tap schedules.
• Container ceilings. datris-tap-runner is limited to 1 CPU, 2 GB of memory (its 512 MB /tmp scratch counts toward it) and 256 processes; datris-codegen-runner to 1 CPU, 1 GB and 128 processes (threads count toward the process ceiling). These are ceilings, not reservations: an idle runner uses no more than before.
• Per-script limits. Each script gets a 1024 MB address-space limit (TAP_RUNNER_SCRIPT_MEMORY_MB, 0 = none). A script that allocates past the limit fails with MemoryError instead of pushing the runner out of memory. The address-space limit is not enforced on macOS when you run the runner outside Docker.
• Native thread pools sized to the runner. Libraries such as numpy (OpenBLAS), OpenMP and MKL start one thread per CPU of the host, whatever the container’s CPU limit. Each script gets OPENBLAS_NUM_THREADS, OMP_NUM_THREADS and MKL_NUM_THREADS set to TAP_RUNNER_SCRIPT_THREADS (default 1), so several pandas taps at once stay well inside the process ceiling. A tap can set its own value through its environment.

​

Configuration

USE_TAP_RUNNER=false sbt -Dconfig.file=... run

​

What changes for tap authors

​

Reading platform data from a tap

import os, json, urllib.request

def fetch():
    host = os.environ["DATRIS_PLATFORM_HOST"]   # "localhost" in-process, "datris" in the runner
    port = os.environ["DATRIS_PLATFORM_PORT"]
    db   = os.environ["DATRIS_POSTGRES_DATABASE"]
    req = urllib.request.Request(
        f"http://{host}:{port}/api/v1/query/postgres",
        data=json.dumps({"sql": "SELECT ticker FROM stock_tickers", "database": db}).encode(),
        headers={"Content-Type": "application/json"},
    )
    tickers = json.loads(urllib.request.urlopen(req).read())["results"]
    # ...use tickers to drive the rest of fetch()...
    return tickers

​

When to use which

• Isolated runner (compose/prod default) — tap code stays separated from platform credentials and internal services. Isolated taps cannot open a direct connection to internal DB / MinIO / Vault; they return records and the pipeline writes destinations.
• In-process (sbt / IDE, or USE_TAP_RUNNER=false) — tap fetch() runs inside the server JVM. Fine for local development without the sidecar, or a legacy tap that still opens a direct internal connection. The server logs a loud warning. Do not treat a laptop compose install as in-process — compose isolates by default.

​

Generated data-quality and transformation scripts

• USE_CODEGEN_RUNNER=true (the compose default): the runner is required. If it is stopped, unreachable, an older image without the /execute-file endpoint, or rejects the token, the rule or transformation fails with a message naming the runner. Datris never falls back to running the script in-process.
• USE_CODEGEN_RUNNER unset or false (sbt / IDE, or an install still on an older compose file): scripts run in-process, with only the scrubbed environment, as before. The server logs a warning at startup and on every run, and datris doctor reports codegen.isolation as warn.