Skip to main content

​
Prerequisites

​
Quick Start

All you need is Docker — no git checkout, no build tools. The installer pulls the pre-built images from Docker Hub, fetches the few runtime files Compose needs into a ./datris directory, seeds a .env, and starts the full stack:
The installer walks you through a short setup — press Enter at every prompt for a standard install:
  1. AI keys. Anthropic (recommended for chat, CodeGen, AI data quality, NL→SQL) and OpenAI (recommended for semantic-search embeddings). Both are optional and each is one Enter to skip. Using Azure OpenAI instead? Skip both prompts and the installer asks for your Azure key, resource endpoint, and chat deployment name next.
  2. Databases and stores. Pick what to run — each can be bundled (a local container), external (point at a service you already have), or skipped:
    • Postgres — structured destination + pgvector (default: bundled)
    • Semantic search embeddings — OpenAI text-embedding-3-small (recommended: no 2.2 GB model download, no resident container) or the bundled local TEI server for data that can’t leave the machine
    • Vector stores — qdrant, weaviate, chroma (default: none)
    • Kafka — local test broker or external (default: none)
    • Snowflake / Databricks — optionally store destination credentials now so pipelines can use them on day one
It prints an install summary, starts the stack, and then checks every store you configured — a mistyped external hostname or credential shows up right there by name, not later mid-pipeline. On first start, vault-init seeds your keys and store credentials into Vault. Changing your mind later is one line in ./datris/.env — for example POSTGRES_ENABLED=0, COMPOSE_PROFILES=qdrant,kafka, or EMBEDDING_PROVIDER=openai — followed by docker compose up -d. Disabling a store removes its container but keeps its data volume, so re-enabling restores the data. The .env.example comments document every knob.
The install.sh installer is a POSIX shell script, so it runs on macOS and Linux. On Windows, run it from a POSIX shell — WSL2 (recommended) or Git Bash — or skip the installer and use the single-file Compose option below, which works natively in PowerShell.
Or, for a fully self-contained single file (no installer) — the init scripts and config are inlined, so nothing else is needed (requires Docker Compose ≥ 2.23). This is the simplest path on Windows:
In PowerShell, use curl.exe (the bundled curl alias maps to Invoke-WebRequest and takes different flags), and set the key with $env: since the inline KEY=value command syntax is bash-only.

​
Verify

​
Alternative: install from a git clone

If you’d rather work from a checked-out repository — for example to track the source, customize docker-compose.yml, or contribute — clone the repo and start from the checked-out compose file:
This still pulls the pre-built images from Docker Hub — no build tools required. To build the images yourself, see Building from Source below.

​
Upgrading

Upgrading means two things: pulling the latest pre-built images and refreshing the Compose file itself, since a new release can rename, add, or drop services. How you do it depends on how you installed. In every case your data is preserved — it lives in Docker volumes that survive an upgrade. Installer (the curl … | sh Quick Start): just re-run the install command. It re-downloads the latest compose file and runtime scripts, leaves your .env and data untouched, pulls the new images, and restarts.
Single-file (docker-compose.standalone.yml): re-download the file (it carries the latest topology and inlined scripts), then pull and restart.
From source (cloned repo): pull the latest compose file from git, then pull images and restart.
No build tools required for any of these — the images come pre-built from Docker Hub.
Always pass --remove-orphans when upgrading manually. (The installer does this for you.) Releases occasionally rename, remove, or repurpose service blocks in docker-compose.yml (e.g. v1.6.15 replaced the bundled Ollama service with TEI on the same host port 11434). Without --remove-orphans, the previous version’s container keeps running and holding the port, causing the new container to fail with Bind for 0.0.0.0:<port> failed: port is already allocated. The flag is safe — it only removes containers that are no longer defined in your current compose file. Volumes (and your data) are untouched.
Your data is preserved across upgrades — Postgres pipelines, Vault secrets, MinIO buckets, MongoDB collections, and Kafka state all live in Docker volumes that survive docker compose pull and docker compose up -d.

​
Stale secrets after an upgrade

Datris occasionally deprecates secret paths between versions (for example, v1.5.6 split AI configuration into three new Vault slots and stopped reading the old single-slot path). Because Vault data persists across upgrades, deprecated entries stay in Vault and continue to appear in the Configuration → Secrets list even though the server ignores them. To clean them up:
  • Targeted (preserves your data): delete the stale entries one at a time from Configuration → Secrets in the UI, or run docker compose exec -e VAULT_TOKEN=root-token vault vault kv delete secret/oss/<name> for each deprecated path.
  • Total reset (destroys all data): see the next section.

​
Reset everything (destroys all data)

If you don’t care about anything on this machine and want a completely fresh install — same data layout as a brand-new clone — wipe all volumes:
Warning: docker compose down -v is destructive. The -v flag removes every Docker volume the project owns (both named volumes and the anonymous volumes the data services create automatically). You will lose:
  • All your pipelines, runs, taps, and metadata (Postgres datris database)
  • All Vault secrets — API keys, database credentials, AI configuration (will be re-seeded from your .env on next start, but only the defaults — any UI-edited overrides are gone)
  • All MinIO object storage — raw uploads, configs, temp files, processed outputs
  • MongoDB destination data
  • Queued messages and offsets in Kafka, Zookeeper, ActiveMQ
  • The bundled bge-m3 embedding model in the tei-data volume (will re-download ~2.2 GB on first start)
  • Cached pip wheels in the pip-cache volume (taps that need extras like yfinance will re-download on first run)
Only run down -v if you are certain none of the above matters, e.g. on a brand-new dev machine, after exporting anything you needed, or on a CI runner. Never run down -v on a production or shared instance. After docker compose up -d, vault-init.sh re-seeds the AI configuration secrets from your .env, MinIO buckets are recreated, Postgres starts empty, and the bundled embedding service re-downloads bge-m3 (a few minutes one-time).
Note for production deployments: the deploy/docker-compose.prod.yml file used by managed/dedicated installs uses bind mounts to host directories under /data/* instead of Docker volumes, so docker compose -f docker-compose.prod.yml down -v does not wipe the data — the host directories survive. To reset a prod install, you’d need to also delete the relevant /data/* directories on the host, which is a much riskier operation and not recommended outside of disaster recovery.

​
Volumes

All stateful services use named Docker volumes, created automatically by docker compose up — no user action required: Because these are named volumes, they re-attach whenever a container is recreated or a store is disabled (POSTGRES_ENABLED=0) and later re-enabled — disable → re-enable is lossless. If you upgraded from a version where the data services used anonymous volumes, the first up -d after this change starts the stores on fresh named volumes; the installer-era data lives on in the old anonymous volumes (visible via docker volume ls -f dangling=true) and can be copied across with docker run --rm -v <old>:/src:ro -v <new>:/dst alpine cp -a /src/. /dst/. Commonly used packages (requests, beautifulsoup4, pandas, lxml, feedparser, boto3, google-cloud-storage, azure-storage-blob, openpyxl, pyyaml, python-dateutil, pytz) are baked into the image. When a tap needs something extra (e.g. yfinance), pip downloads it on first run (~30 seconds) and caches the wheel in pip-cache. Subsequent container restarts re-run pip install for those extras, but the install is near-instant because the wheel is already cached locally.

​
Services

Optional vs opt-in. Services that ship enabled (Postgres, TEI) are disabled with an explicit *_ENABLED=0 in .env — an .env without these lines keeps them running, so upgrades never silently drop a service you were using. Brand-new services (vector stores, Kafka) activate via COMPOSE_PROFILES. Datris bundles kafka-clients, so pipelines that talk to an external Kafka don’t need the local broker at all — set KAFKA_BOOTSTRAP_SERVERS in .env instead. One teardown nuance: docker compose down ignores inactive profiles, so for a full teardown use docker compose --profile "*" down.

​
Web UIs

​
API Keys and AI Providers

Datris supports four AI providers. Set your keys in .env: At least one AI provider key is required for AI features. The embedding provider for RAG is configured via Vault secrets — see AI Configuration for details.

​
Infrastructure Details

​
MinIO

The minio-init container automatically creates the required buckets:
  • {env}-raw - File upload staging
  • {env}-raw-plus - Processed file staging
  • {env}-temp - Temporary processing files
  • {env}-data - Pipeline output (object store destination)
  • {env}-config - Configuration files (validation schemas)
Where {env} is the environment name (default: oss). See Configuration Reference for the environment setting.

​
Vault

On first boot, the vault-init container seeds Vault with default secrets for the bundled services (MinIO, ActiveMQ, MongoDB, PostgreSQL) plus your AI provider API keys and any external-store credentials (Kafka, vector stores, Snowflake, Databricks) from .env. Vault uses durable file storage on the vault-data volume, so secrets — including any you add later in the Configuration tab or via taps — persist across restarts and rebuilds. .env is the first-boot seed only; after that the Configuration tab is the source of truth. See How Configuration Persists for details and the clean-reset path.

​
Vector Databases

pgvector is included whenever Postgres is enabled (bundled or external). The others are one line in .env:
  • Qdrant — high-performance vector database: COMPOSE_PROFILES=qdrant
  • Weaviate — open-source vector database: COMPOSE_PROFILES=weaviate
  • Chroma — lightweight, single container: COMPOSE_PROFILES=chroma
  • Milvus — scalable vector database; runs external-only (it needs its own etcd/minio stack — see the Milvus standalone install), then set MILVUS_HOST/MILVUS_PORT in .env
Combine profiles with commas: COMPOSE_PROFILES=qdrant,kafka. To use a managed/cloud vector store instead of a bundled one, skip the profile and set the store’s *_HOST / *_PORT / *_API_KEY variables in .env (first boot seeds the credentials into Vault; afterwards manage them in Configuration → Secrets).

​
External databases and stores

Every optional store can point at infrastructure you already run instead of a bundled container — the installer asks, or set the variables in .env before first boot: POSTGRES_JDBC_URL/POSTGRES_USER/POSTGRES_PASSWORD (with POSTGRES_ENABLED=0), KAFKA_BOOTSTRAP_SERVERS, vector-store *_HOST variables, and SNOWFLAKE_* / DATABRICKS_* destination credentials. These are first-boot seeds: vault-init writes them into Vault create-if-absent, and from then on Configuration → Secrets is the source of truth. The platform’s availability probes treat external stores exactly like bundled ones — the pipeline wizard and the Assistant offer whatever is actually reachable.

​
Configuration

The pipeline server reads configuration from application.yaml, mounted from docker/config/application.yaml. See Configuration Reference for the full list of properties.

​
JVM Heap Sizing

The datris service runs a Spring Boot JVM. Its heap is governed by the JAVA_OPTS environment variable, passed in via docker-compose.yml. The default is sized to fit comfortably on an 8 GB host alongside the bundled TEI embedder, Postgres, MongoDB, MinIO, ActiveMQ, Vault, the UI, and the MCP server:
Override in your .env file on larger hosts. Suggested sizings: Example for a 24 GB host:
Then restart the service:
Docker Desktop note (macOS / Windows): Docker Desktop runs containers inside a Linux VM with its own RAM allocation, which may be much lower than your host’s total RAM. Check the VM ceiling with:
If the VM is capped below what you need, raise it in Docker Desktop → Settings → Resources → Memory. The JVM -Xmx must be smaller than the Docker VM’s allocation, otherwise the kernel inside the VM will OOM-kill containers under load.

​
Building from Source

For development or contributing:

​
Prerequisites

​
Build and run

In docker-compose.yml, uncomment the build: lines and comment out the image: lines for the services you want to build locally: