Skip to main content
This reference lists every data type the pipeline supports, along with mappings to PostgreSQL and Spark types used during storage and transformation. The external warehouse and database destinations have their own DDL mappings from the same pipeline types — see the type-mapping tables on the Snowflake, Databricks and SQL Server destination pages.

​
Type Reference

The PostgreSQL Type column shows the literal type Datris emits in the generated CREATE TABLE DDL. Datris remaps only a few types to their canonical Postgres aliases (tinyint/smallint → int2, float → float4, double → float8, string → text); every other type is emitted verbatim.

​
Precision and Scale for Decimal

The decimal(p,s) type requires two parameters:
  • p (precision): total number of digits, range 1 to 38.
  • s (scale): number of digits to the right of the decimal point, range 0 to p.
Examples:
A pipeline that uses the deprecated transformation.rowFunctions (JavaScript or REST row functions) converts every decimal column to a 64-bit double on the way into the function and writes the result back as text, so digits beyond IEEE double precision (about 15–17 significant digits) are lost before the value reaches the destination, whatever (p,s) declares. Pipelines without row functions are not affected: values stay as the text that was ingested until the destination casts them (Spark/object store uses DecimalType(p,s); PostgreSQL COPY, Snowflake, Databricks and SQL Server receive the declared decimal(p,s) type). If you need exact values above that precision and must keep a row function, declare the field as string — it lands as text in PostgreSQL, VARCHAR in Snowflake and NVARCHAR in SQL Server — and cast it downstream.

​
Integer Type Selection

Choose the narrowest integer type that fits your data to reduce storage and improve Spark performance:

​
String Type Selection

All three string types map to StringType in Spark. The length constraint is enforced only at the PostgreSQL layer.

​
JSON and XML Special Types

JSON and XML pipelines are semi-structured: the source schema must contain exactly one field, named literally _json or _xml (matching the source file type), with type string. The validator rejects any schema with a different field count or a different field name. Declare the single field with type string in the schema:
For XML, use the same shape with a single _xml field. During ingestion, the pipeline preserves the raw content without attempting to parse it into individual columns. This is useful for semi-structured data that should be queried with JSON or XML functions downstream.

​
Type Coercion

During ingestion, the pipeline converts field values from strings to the declared type:
  1. Null or empty string — passed through unchanged (no type conversion is applied). The Postgres COPY layer then converts an empty field to NULL.
  2. Numeric types (int, bigint, float, double, decimal, etc.) — the value is parsed using Scala’s built-in conversion. decimal is parsed via toDouble in the row-map path. If parsing fails (e.g., "abc" in an int column), the job fails with an error.
  3. String types (string, varchar, char) — stored as-is with no conversion.
  4. Date and timestamp — stored as the raw string value. The destination (PostgreSQL, Spark) handles date/timestamp parsing according to its own rules.