Skip to main content
Use this destination when your lake lives in S3 and you want validated, lineage-tracked files landed there by an agent or a scheduled pipeline, with credentials scoped per bucket. The S3 destination writes Parquet files, ORC files, or an Apache Iceberg table directly to AWS S3. It uses the same Spark + S3A code path as the MinIO destination; the only differences are the credentials, the endpoint, and a few S3A settings applied per-bucket so multiple S3 destinations can coexist with the built-in MinIO output.

​
How it differs from MinIO

S3A’s per-bucket key syntax (fs.s3a.bucket.<bucket>.*) is what allows a single Spark session to write to both MinIO and AWS S3 in the same deployment without one set of credentials clobbering the other.

​
Credentials secret

Region lives in the credentials secret next to accessKey and secretKey. Bundling region with the key it pairs with keeps the credential self-contained and prevents the silent-failure mode where the pipeline config claims us-east-1 but the key is scoped to us-west-2. Create a Vault secret (Secrets tab in the Datris UI) with these fields: If any of accessKey, secretKey, or region is missing, the write fails at resolve time with an explicit “missing required field” error naming the secret.

​
Rotating credentials

To rotate or correct the keys, update the secret in the Secrets tab. Fields you leave masked keep their current value, so you can change region or add sessionToken without re-entering accessKey and secretKey. The next run of any pipeline that uses that secret picks up the new keys; no restart is needed. A query against the same bucket that is already running at the moment the keys change can fail with “Filesystem closed” — re-run it.

​
Configuration

​
Field reference

* The credentialsSecret field is optional only when Datris itself runs in AWS with an instance role attached; the AWS default credentials provider chain will pick up credentials and region from instance metadata. For self-hosted Datris (not running on EC2 / EKS / ECS), credentialsSecret is effectively required.

​
File formats, write modes, partitioning, type casting

These behave the same as the MinIO destination — see the Object Store Destination page for the full reference, and the Iceberg page for table commits, merge, and schema evolution. To make an Iceberg table written to S3 findable by name in Databricks, register it in Unity Catalog; see Register Iceberg tables.

​
Multiple S3 destinations

Per-bucket credential isolation means two pipelines can write to different S3 buckets using different IAM keys in the same deployment. Each pipeline references its own credentialsSecret; the credentials are applied to that specific bucket only.