Skip to main content

​
Prerequisites

  • Docker
  • Memory: 4 GB available to Docker for the minimal install, about 8 GB for the full stack with the bundled embedding server. On Docker Desktop this is the VM allocation under Settings → Resources, not your host’s total RAM.
License. Datris is open source under the Apache License 2.0. Run it, modify it, embed it, or host it for others; the license asks only that you keep the license and NOTICE files with any copy you redistribute and mark the files you changed. Commercial support is available — see Licensing & Support.Telemetry. The platform sends no usage data, crash reports, or license checks anywhere. The two outbound requests it can make are an on-demand model-catalog refresh from the Configuration tab, which falls back to a built-in list when it fails, and the datris doctor update check, which sends only the server version, operating system and CPU architecture to datris.ai when you run datris doctor (add DATRIS_UPDATE_CHECK=0 to .env to switch it off) — see Security Architecture for how to verify.

​
Quick Start

All you need is Docker and one AI provider key (Anthropic, OpenAI, Grok, Azure OpenAI, or Amazon Bedrock) — no git checkout, no build tools. The installer pulls the pre-built images from Docker Hub, fetches the few runtime files Compose needs into a ./datris directory, seeds a .env, and starts the full stack:
Installing from a coding agent or a CI job with no terminal attached? Export the key first (ANTHROPIC_API_KEY, OPENAI_API_KEY, or another provider’s): the installer asks nothing and reads it from the environment. Full steps: Install for Agents. Installing for production? Add DATRIS_GOVERNED=1 (curl -fsSL https://get.datris.ai/install.sh | DATRIS_GOVERNED=1 sh) to switch on user login, API keys, the audit log and the agent policy in the new .env; for an existing install see Switch on the governance controls.

​
Minimal install

Trying Datris on a laptop? The default stack runs about ten containers and the bundled embedding server (TEI) downloads a 2.2 GB model on first boot. None of that is required. Three settings cut it to nine small containers, no model download, and roughly 3 GB of memory at idle:
The installer applies the first two for you when you pick OpenAI embeddings, and the single-file Compose path below defaults to OpenAI embeddings whenever an OPENAI_API_KEY is set. Everything here is reversible: flip a value back to 1, run docker compose up -d, and the service returns with its data volume intact. The datris server itself runs on a 2 GB heap by default; see JVM Heap Sizing. The installer walks you through a short setup — press Enter at every prompt for a standard install:
  1. AI keys. Datris can use Anthropic Claude, OpenAI, or Grok (xAI) — the installer prompts for each key in turn; enter any you have (one is enough, each prompt is one Enter to skip). If you prefer Claude through Amazon Bedrock or OpenAI through Azure OpenAI, skip the key prompts and the installer offers those routes next — Azure asks for your key, resource endpoint, and chat deployment name; Bedrock asks for AWS credentials (or Enter to use the host’s IAM role) and a region. When you enter more than one key, the installer asks which provider handles chat and CodeGen; you can change it later in the Configuration tab. The stack needs one AI provider to start, so the installer stops if none is given.
  2. Databases and stores. Pick what to run — each can be bundled (a local container), external (point at a service you already have), or skipped:
    • Postgres — structured destination + pgvector (default: bundled)
    • Semantic search embeddings — OpenAI text-embedding-3-small (recommended: no 2.2 GB model download, no resident container) or the bundled local TEI server for data that can’t leave the machine
    • Vector stores — pgvector by default (included with Postgres, no extra container); optionally add qdrant, weaviate, or chroma
    • Kafka — local test broker or external (default: none)
    • Snowflake / Databricks / SQL Server — optionally store destination credentials now so pipelines can use them on day one (SQL Server: a SQL login, Microsoft Entra ID, or Kerberos)
It prints an install summary, starts the stack, and then checks every store you configured — a mistyped external hostname or credential shows up right there by name, not later mid-pipeline. On first start, vault-init seeds your keys and store credentials into Vault. Changing your mind later is one line in ./datris/.env — for example POSTGRES_ENABLED=0, COMPOSE_PROFILES=qdrant,kafka, or EMBEDDING_PROVIDER=openai — followed by docker compose up -d. Disabling a store removes its container but keeps its data volume, so re-enabling restores the data. The .env.example comments document every knob.
The install.sh installer is a POSIX shell script, so it runs on macOS and Linux. On Windows, run it from a POSIX shell — WSL2 (recommended) or Git Bash — or skip the installer and use the single-file Compose option below, which works natively in PowerShell.
Or, for a fully self-contained single file (no installer) — the init scripts and config are inlined, so nothing else is needed (requires Docker Compose ≥ 2.23). This is the simplest path on Windows:
The bundled embedding server (TEI) is optional and starts by default, pulling a 2.2 GB model on first boot. To skip it, add TEI_ENABLED=0 — and, if you have an OpenAI key, OPENAI_API_KEY=sk-... — to the environment or a .env next to the file. With an OpenAI key present the standalone file uses OpenAI embeddings automatically; without one, semantic search is off until you enable an embedding provider in the Configuration tab. In PowerShell, use curl.exe (the bundled curl alias maps to Invoke-WebRequest and takes different flags), and set the key with $env: since the inline KEY=value command syntax is bash-only.

​
Verify

​
Alternative: install from a git clone

If you’d rather work from a checked-out repository — for example to track the source, customize docker-compose.yml, or contribute — clone the repo and start from the checked-out compose file:
This still pulls the pre-built images from Docker Hub — no build tools required. To build the images yourself, see Building from Source below.

​
Upgrading

Upgrading pulls the latest images and refreshes the Compose file, and your data survives in Docker volumes. The procedure for each install method, the pre-upgrade self-check, and what to do about stale secrets live on Upgrades & Supported Versions.

​
Reset everything (destroys all data)

If you don’t care about anything on this machine and want a completely fresh install — same data layout as a brand-new clone — wipe all volumes:
Warning: docker compose down -v is destructive. The -v flag removes every Docker volume the project owns (all of them are named volumes; see Volumes). You will lose:
  • All your pipelines, runs, taps, and metadata (MongoDB internal oss database) and all data landed in Postgres (datris database)
  • All Vault secrets — API keys, database credentials, AI configuration (will be re-seeded from your .env on next start, but only the defaults — any UI-edited overrides are gone)
  • All MinIO object storage — raw uploads, configs, temp files, processed outputs
  • MongoDB destination data
  • Queued messages and offsets in Kafka, Zookeeper, ActiveMQ
  • The bundled bge-m3 embedding model in the tei-data volume (will re-download ~2.2 GB on first start)
  • Cached pip wheels in the pip-cache volume (used only when taps run in-process with USE_TAP_RUNNER=false)
Only run down -v if you are certain none of the above matters, e.g. on a brand-new dev machine, after exporting anything you needed, or on a CI runner. Never run down -v on a production or shared instance. After docker compose up -d, vault-init.sh re-seeds the AI configuration secrets from your .env, MinIO buckets are recreated, Postgres starts empty, and the bundled embedding service re-downloads bge-m3 (a few minutes one-time).
Note for production deployments: the optional deploy/docker-compose.prod.yml layout uses bind mounts to host directories under /data/* instead of Docker volumes, so docker compose -f docker-compose.prod.yml down -v does not wipe the data — the host directories survive. To reset a prod install, you’d need to also delete the relevant /data/* directories on the host, which is a much riskier operation and not recommended outside of disaster recovery.

​
Volumes

All stateful services use named Docker volumes, created automatically by docker compose up — no user action required: Because these are named volumes, they re-attach whenever a container is recreated or a store is disabled (POSTGRES_ENABLED=0) and later re-enabled — disable → re-enable is lossless. If you upgraded from a version where the data services used anonymous volumes, the first up -d after this change starts the stores on fresh named volumes; the installer-era data lives on in the old anonymous volumes (visible via docker volume ls -f dangling=true) and can be copied across with docker run --rm -v <old>:/src:ro -v <new>:/dst alpine cp -a /src/. /dst/. Commonly used packages (requests, beautifulsoup4, pandas, lxml, feedparser, boto3, google-cloud-storage, azure-storage-blob, openpyxl, pyyaml, python-dateutil, pytz) are baked into the image. When a tap needs something extra (e.g. yfinance), the isolated tap runner (the Compose default) installs it into a throwaway environment for that run — see Tap Execution & Isolation. With USE_TAP_RUNNER=false the in-process path installs it on first run (~30 seconds) and caches the wheel in pip-cache, so later installs are near-instant.

​
Services

Ports marked localhost only are bound to 127.0.0.1 in docker-compose.yml, so MinIO, MongoDB, PostgreSQL and Vault are reachable from the Docker host but not from other machines. Every other port listed binds to all interfaces.
Optional vs opt-in. Services that ship enabled (Postgres, TEI) are disabled with an explicit *_ENABLED=0 in .env — an .env without these lines keeps them running, so upgrades never silently drop a service you were using. Brand-new services (vector stores, Kafka) activate via COMPOSE_PROFILES. Datris bundles kafka-clients, so pipelines that talk to an external Kafka don’t need the local broker at all — set KAFKA_BOOTSTRAP_SERVERS in .env instead. One teardown nuance: docker compose down ignores inactive profiles, so for a full teardown use docker compose --profile "*" down.

​
Web UIs

​
API Keys and AI Providers

Datris supports six AI providers. Set your keys in .env: At least one AI provider key is required for AI features. The embedding provider for RAG is configured via Vault secrets — see AI Configuration for details.

​
Infrastructure Details

​
MinIO

The minio-init container automatically creates the required buckets:
  • {env}-raw - File upload staging
  • {env}-temp - Temporary processing files
  • {env}-data - Pipeline output (object store destination)
  • {env}-config - Configuration files (validation schemas)
Where {env} is the environment name (default: oss). See Configuration Reference for the environment setting.

​
Vault

On first boot, the vault-init container seeds Vault with default secrets for the bundled services (MinIO, ActiveMQ, MongoDB, PostgreSQL) plus your AI provider API keys and any external-store credentials (Kafka, vector stores, Snowflake, Databricks, SQL Server) from .env. Vault uses durable file storage on the vault-data volume, so secrets — including any you add later in the Configuration tab or via taps — persist across restarts and rebuilds. .env is the first-boot seed only; after that the Configuration tab is the source of truth. See How Configuration Persists for details and the clean-reset path.

​
Vector Databases

pgvector is included whenever Postgres is enabled (bundled or external). The others are one line in .env:
  • Qdrant — high-performance vector database: COMPOSE_PROFILES=qdrant
  • Weaviate — open-source vector database: COMPOSE_PROFILES=weaviate
  • Chroma — lightweight, single container: COMPOSE_PROFILES=chroma
  • Milvus — scalable vector database; runs external-only (it needs its own etcd/minio stack — see the Milvus standalone install), then set MILVUS_HOST/MILVUS_PORT in .env
Combine profiles with commas: COMPOSE_PROFILES=qdrant,kafka. To use a managed/cloud vector store instead of a bundled one, skip the profile and set the store’s *_HOST / *_PORT / *_API_KEY variables in .env (first boot seeds the credentials into Vault; afterwards manage them in Configuration → Secrets).

​
External databases and stores

Every optional store can point at infrastructure you already run instead of a bundled container — the installer asks, or set the variables in .env before first boot: POSTGRES_JDBC_URL/POSTGRES_USER/POSTGRES_PASSWORD (with POSTGRES_ENABLED=0), KAFKA_BOOTSTRAP_SERVERS, vector-store *_HOST variables, and SNOWFLAKE_* / DATABRICKS_* / SQLSERVER_* destination credentials (see SQL Server for the SQLSERVER_* names per auth method; the installer seeds password-based Kerberos only, so create a keytab Kerberos secret under Configuration → Secrets → Platform). These are first-boot seeds: vault-init writes them into Vault create-if-absent, and from then on Configuration → Secrets is the source of truth. The platform’s availability probes treat external stores exactly like bundled ones — the pipeline wizard and the Assistant offer whatever is actually reachable.

​
Configuration

The pipeline server reads configuration from application.yaml, mounted from docker/config/application.yaml. See Configuration Reference for the full list of properties.

​
JVM Heap Sizing

The datris service runs a Spring Boot JVM. Its heap is governed by the JAVA_OPTS environment variable, passed in via docker-compose.yml. The default is a 2 GB maximum heap:
That default is the same for every install size. What changes the memory footprint is which optional services run alongside it: Configured limits is the sum of the memory ceilings set in docker-compose.yml — the datris JVM heap (-Xmx2g), datris-tap-runner (mem_limit 2 GB) and datris-codegen-runner (mem_limit 1 GB); every other service runs without a limit, so it is not a hard cap on the stack. Measured is docker stats resident memory of an idle stack (datris about 1.4 GB, Vault 0.4, MongoDB 0.4, ActiveMQ 0.3, MinIO 0.2, UI + MCP server 0.1, the two runners under 0.1 between them, Postgres under 0.1, TEI about 5 GB); actual use sits between the two, since the runners only approach their limits while scripts run. So plan on 4 GB of Docker memory for a minimal or default install without TEI (the measured 3 GB plus room for the JVM to fill its 2 GB heap), and at least 8 GB when the bundled embedding server runs. Override JAVA_OPTS in your .env file on larger hosts if you hit OutOfMemoryError during heavy ingestion, embedding batches, or large query results. Suggested sizings: Example for a 24 GB host:
Then restart the service:
Docker Desktop note (macOS / Windows): Docker Desktop runs containers inside a Linux VM with its own RAM allocation, which may be much lower than your host’s total RAM. Check the VM ceiling with:
If the VM is capped below what you need, raise it in Docker Desktop → Settings → Resources → Memory. The JVM -Xmx must be smaller than the Docker VM’s allocation, otherwise the kernel inside the VM will OOM-kill containers under load. Runner limits. The two script runners carry their own ceilings so a burst of taps or AI rules cannot exhaust the host: datris-tap-runner at 1 CPU / 2 GB / 256 processes and datris-codegen-runner at 1 CPU / 1 GB / 128 processes, each running at most 4 scripts at once (further runs wait their turn). These are limits, not reservations, so the memory figures above do not grow. On a larger host that runs many taps on overlapping schedules, raise them together in .env: See Overload protection for every setting.

​
Building from Source

For development or contributing:

​
Prerequisites

  • JDK 17 (the build targets --release 17; newer JDKs are not tested)
  • SBT

​
Build and run

In docker-compose.yml, uncomment the build: lines and comment out the image: lines for the services you want to build locally: