Skip to main content
This reference lists every data type the pipeline supports, along with mappings to PostgreSQL and Spark types used during storage and transformation. The external warehouse destinations have their own DDL mappings from the same pipeline types — see the type-mapping tables on the Snowflake and Databricks destination pages.

​
Type Reference

The PostgreSQL Type column shows the literal type Datris emits in the generated CREATE TABLE DDL. Datris remaps only a few types to their canonical Postgres aliases (tinyint/smallint → int2, float → float4, double → float8, string → text); every other type is emitted verbatim.

​
Precision and Scale for Decimal

The decimal(p,s) type requires two parameters:
  • p (precision): total number of digits, range 1 to 38.
  • s (scale): number of digits to the right of the decimal point, range 0 to p.
Examples:

​
Integer Type Selection

Choose the narrowest integer type that fits your data to reduce storage and improve Spark performance:

​
String Type Selection

All three string types map to StringType in Spark. The length constraint is enforced only at the PostgreSQL layer.

​
JSON and XML Special Types

JSON and XML pipelines are semi-structured: the source schema must contain exactly one field, named literally _json or _xml (matching the source file type), with type string. The validator rejects any schema with a different field count or a different field name. Declare the single field with type string in the schema:
For XML, use the same shape with a single _xml field. During ingestion, the pipeline preserves the raw content without attempting to parse it into individual columns. This is useful for semi-structured data that should be queried with JSON or XML functions downstream.

​
Type Coercion

During ingestion, the pipeline converts field values from strings to the declared type:
  1. Null or empty string — passed through unchanged (no type conversion is applied). The Postgres COPY layer then converts an empty field to NULL.
  2. Numeric types (int, bigint, float, double, decimal, etc.) — the value is parsed using Scala’s built-in conversion. decimal is parsed via toDouble in the row-map path. If parsing fails (e.g., "abc" in an int column), the job fails with an error.
  3. String types (string, varchar, char) — stored as-is with no conversion.
  4. Date and timestamp — stored as the raw string value. The destination (PostgreSQL, Spark) handles date/timestamp parsing according to its own rules.