API File Upload

Upload data files directly to the pipeline through the REST API. The pipeline accepts any file type (CSV, JSON, XML, Excel, PDF, Word, PowerPoint, HTML, email, EPUB, plain text, and archives), handles decompression automatically, and returns a tracking token.

Endpoint

POST /api/v1/pipeline/upload

Request

The request uses multipart/form-data encoding with the following parts:

Part	Required	Description
`file`	Yes	The data file to upload
`pipeline`	Yes	Name of the target pipeline configuration
`publishertoken`	No	Opaque token identifying the data publisher; used for lineage tracking

Example

curl -X POST "http://localhost:8080/api/v1/pipeline/upload" \
  -F "file=@transactions_2025.csv" \
  -F "pipeline=transactions" \
  -F "publishertoken=finance-team-a"

Response

A successful upload returns HTTP 200 with a bare string body — the pipeline token itself (a unique UUID generated for the job), not a JSON object:

d4f8e1a2-7b3c-4e9f-a5d6-1c2b3e4f5a6b

Use this token to track the job’s processing status, view errors, or cancel the job. When a compressed archive is processed on the per-file path (non-CSV pipelines, or a single-file archive), the response is instead the literal string:

3 file(s) submitted

Compressed Files

When the uploaded file has a compressed extension, the archive is extracted in memory and processed synchronously — nothing is staged to MinIO:

Extension	Handling
`.zip`	Extracted in memory
`.gz`	Decompressed in memory (gunzip only — a `.tar.gz` is not untarred; the inner `.tar` is submitted as a single file)
`.tar`	Extracted in memory
`.jar`	Extracted in memory

Compressed archives may contain multiple data files. Handling depends on the pipeline type:

CSV pipelines — when an archive contains multiple inner files, they are concatenated into a single job. The header is kept from the first file and stripped from the 2nd and later files. The response is the single pipeline token.
Non-CSV pipelines, or single-file archives — each inner file is submitted as its own job. The response is always the "N file(s) submitted" string on this path, even for a single file (1 file(s) submitted) — the pipeline token is not returned.

Uncompressed Files

Files without a recognized compressed extension (e.g., .csv, .json, .xml, .xls, .pdf, .docx, .txt, etc) are read into memory and submitted directly. They are not staged to MinIO.

Processing Flow

The client sends the multipart request.
The endpoint validates that the named pipeline is registered.
The file bytes are read into memory and the extension is inspected.
Compressed path: the archive is decompressed in memory. For CSV pipelines with multiple inner files, the files are concatenated (headers stripped after the first) and submitted as one job; otherwise each inner file is submitted individually.
Uncompressed path: the file contents are passed directly to the ingestion pipeline.
The endpoint returns the pipeline token (or the "N file(s) submitted" string).

Error Responses

HTTP Status	Cause
400	Missing `pipeline` request parameter (Spring binding error)
500	Pipeline not registered, empty file, internal error, or processing failure

The 500 response covers all errors handled by the controller, including an empty file and an unregistered pipeline. There is no explicit 413 handler; exceeding the configured multipart size limit produces Spring’s default multipart-size error.

Size Limits

The maximum upload size is controlled by spring.servlet.multipart.max-file-size in application.yaml. The default is 1 GB. Adjust this value if your files exceed the limit:

spring:
  servlet:
    multipart:
      max-file-size: 1GB

End-to-end example: CSV to PostgreSQL

1. Start the stack

If you haven’t installed yet, follow Installation — the one-command installer gets the stack running in a ./datris directory. Already installed? Start it from your install directory:

docker compose up -d

Wait for the datris container to show Started Application.

2. Register a pipeline

Create a pipeline configuration that defines the source schema and PostgreSQL destination:

curl -X POST http://localhost:8080/api/v1/pipeline \
  -H "Content-Type: application/json" \
  -d '{
    "name": "employees",
    "source": {
      "schemaProperties": {
        "fields": [
          {"name": "id", "type": "int"},
          {"name": "first_name", "type": "string"},
          {"name": "last_name", "type": "string"},
          {"name": "email", "type": "string"},
          {"name": "salary", "type": "double"}
        ]
      },
      "fileAttributes": {
        "csvAttributes": {
          "delimiter": ",",
          "header": true,
          "encoding": "UTF-8"
        }
      }
    },
    "destination": {
      "database": {
        "dbName": "datris",
        "schema": "public",
        "table": "employees",
        "usePostgres": true
      }
    }
  }'

3. Upload a CSV file

Create a sample CSV file:

cat > /tmp/employees.csv << 'EOF'
id,first_name,last_name,email,salary
1,John,Doe,john@example.com,75000.00
2,Jane,Smith,jane@example.com,82000.00
3,Bob,Johnson,bob@example.com,68000.00
EOF

Upload it:

curl -X POST http://localhost:8080/api/v1/pipeline/upload \
  -F "file=@/tmp/employees.csv" \
  -F "pipeline=employees"

The response body is the pipelineToken for tracking — a plain UUID:

5b2f4a1d-8c7e-4f0a-9b3d-6e1c2a4f8b9e

4. Check status

curl "http://localhost:8080/api/v1/pipeline/status?pipelinetoken=5b2f4a1d-8c7e-4f0a-9b3d-6e1c2a4f8b9e"

The status shows processing stages: begin, processing, end.

5. Verify the data

Query the data via the REST API:

curl -X POST http://localhost:8080/api/v1/query/postgres \
  -H "Content-Type: application/json" \
  -d '{"sql": "SELECT * FROM public.employees"}'

The default database is datris — this is where all PostgreSQL pipelines write unless you specify a different dbName in the pipeline’s destination config.

Getting Started

Connect your agent

Assistant

Catalog

Taps

Ingestion

Destinations

Data Quality

Transformation

AI Features

Configuration

Governance & Security

Production

Integrations

Examples

API File Upload

Endpoint

Request

Example

Response

Compressed Files

Uncompressed Files

Processing Flow

Error Responses

Size Limits

End-to-end example: CSV to PostgreSQL

1. Start the stack

2. Register a pipeline

3. Upload a CSV file

4. Check status

5. Verify the data

​Endpoint

​Request

​Example

​Response

​Compressed Files

​Uncompressed Files

​Processing Flow

​Error Responses

​Size Limits

​End-to-end example: CSV to PostgreSQL

​1. Start the stack

​2. Register a pipeline

​3. Upload a CSV file

​4. Check status

​5. Verify the data

Endpoint

Request

Example

Response

Compressed Files

Uncompressed Files

Processing Flow

Error Responses

Size Limits

End-to-end example: CSV to PostgreSQL

1. Start the stack

2. Register a pipeline

3. Upload a CSV file

4. Check status

5. Verify the data