Write Parquet, ORC, or Apache Iceberg tables to MinIO object storage

Use this destination when you want pipeline output landed as columnar files beside your lake rather than in a database. This is the default file destination that ships with Datris. The object store destination uses an embedded Apache Spark engine to write columnar files to MinIO-compatible object storage. Output is written as loose Parquet files, loose ORC files, or an Apache Iceberg table at an S3A path.

This page covers the MinIO (default) provider — the built-in object store that ships with Datris. To write to AWS S3, see S3 Destination — same code path, same file formats, just per-pipeline credentials instead of the global MinIO ones.

Output Path

Files are written to:

s3a://{bucket}/{prefixKey}

The default bucket is {environment}-data. You can override this per pipeline with the destinationBucketOverride field.

Restricting bucket overrides

By default the override is unrestricted: any pipeline can name any bucket on the built-in MinIO, which is reached with the platform’s own MinIO credentials. On a shared MinIO that lets one pipeline read, overwrite, or deleteBeforeWrite another environment’s data. To restrict it, set DATRIS_OBJECTSTORE_BUCKET_ALLOWLIST in .env to a comma-separated list of the buckets pipelines may target:

DATRIS_OBJECTSTORE_BUCKET_ALLOWLIST=analytics-data,archive-data

With the list set, saving a pipeline whose override is not listed is rejected, and a pipeline saved before the list existed is refused when it runs, is queried, or has its data deleted. The default bucket {environment}-data is always allowed even when not listed, so you cannot lock yourself out. Buckets on AWS S3 (provider: s3) are your own and are not checked. datris doctor reports pipelines that use an override without a list, and any stored override that falls outside the list you set.

File Formats

Format	Description
`parquet`	Apache Parquet columnar format (default)
`orc`	Apache ORC columnar format
`iceberg`	Apache Iceberg table at the prefix: atomic commits, `merge` upserts on `keyFields`, schema evolution. See Iceberg

Set the format with the fileFormat field. parquet and orc write loose files under the prefix; iceberg writes a table there that other engines read by path.

Write Modes

Mode	Behavior
`append`	Append new files to the output path (default)
`overwrite`	Replace all existing files at the output path
`ignore`	Do nothing if data already exists at the output path
`errorifexists`	Fail the job if data already exists at the output path
`merge`	Upsert on `keyFields`. Only valid with `fileFormat: iceberg`

Partitioning

Partition output files by one or more columns using the partitionBy array. Spark creates a directory structure based on the distinct values of the specified columns.

Delete Before Write

Set deleteBeforeWrite to true to remove all existing objects under the output prefix before writing new data. This is useful when you need a clean target path but want finer control than the overwrite write mode provides.

Type Casting

Column types from the source schema are mapped to Spark types before writing:

Schema Type	Spark Type
`tinyint`	`ByteType`
`smallint`	`ShortType`
`int`	`IntegerType`
`bigint`	`LongType`
`float`	`FloatType`
`double`	`DoubleType`
`decimal(p,s)`	`DecimalType(p,s)` (scale defaults to 0 if omitted)
`string`	`StringType`
`boolean`	`BooleanType`
`date`	`DateType`
`timestamp`	`TimestampType`

Configuration Example

{
  "name": "sales_pipeline",
  "source": { "..." : "..." },
  "destination": {
    "objectStore": {
      "fileFormat": "parquet",
      "writeMode": "append",
      "prefixKey": "sales/daily",
      "destinationBucketOverride": "analytics-data",
      "partitionBy": ["region", "year"],
      "deleteBeforeWrite": false
    }
  }
}

Field Reference

Field	Required	Default	Description
`prefixKey`	yes		Key prefix under the bucket
`fileFormat`	no	`parquet`	`parquet`, `orc`, or `iceberg`
`writeMode`	no	`append`	`append`, `overwrite`, `ignore`, `errorifexists`, or `merge` (iceberg only)
`keyFields`	with `merge`		Natural-key columns for `writeMode: merge` on an Iceberg table
`destinationBucketOverride`	no	`{environment}-data`	Override the default bucket name
`partitionBy`	no		Array of column names to partition by
`deleteBeforeWrite`	no	`false`	Delete existing objects before writing

Completion Notification

A pipeline notification is published to ActiveMQ on completion. See Notifications for details.

Getting Started

Connect your agent

Assistant

Catalog

Taps

Ingestion

Destinations

Data Quality

Transformation

AI Features

Configuration

Governance & Security

Production

Integrations

Recipes

Examples

Write Parquet, ORC, or Apache Iceberg tables to MinIO object storage

Output Path

Restricting bucket overrides

File Formats

Write Modes

Partitioning

Delete Before Write

Type Casting

Configuration Example

Field Reference

Completion Notification

​Output Path

​Restricting bucket overrides

​File Formats

​Write Modes

​Partitioning

​Delete Before Write

​Type Casting

​Configuration Example

​Field Reference

​Completion Notification

Output Path

Restricting bucket overrides

File Formats

Write Modes

Partitioning

Delete Before Write

Type Casting

Configuration Example

Field Reference

Completion Notification