AI Transformation

​

Configuration

"transformation": {
    "aiTransformation": {
        "instruction": "convert all date values from YYYY-MM-DD format to MM/DD/YYYY format"
    }
}

​

Examples

"aiTransformation": {
    "instruction": "convert all date values from YYYY-MM-DD format to MM/DD/YYYY format"
}

"aiTransformation": {
    "instruction": "standardize the phone_number column to +1-XXX-XXX-XXXX format"
}

"aiTransformation": {
    "instruction": "add a 'sector' column at the end based on the stock symbol, categorizing each as one of: Technology, Healthcare, Finance, Energy, Consumer, Industrial, Other"
}

"aiTransformation": {
    "instruction": "convert dates to YYYY/MM/DD. Trim leading/trailing whitespace from all columns. Remove duplicate rows."
}

​

How it works

1. When the pipeline is saved, Datris combines the source schema (delimiter, column names and types) with your instruction and sends a single prompt to the AI model. When a run has to generate the script instead (see When the script is generated), it uses the column names and sample rows from that run’s data.
2. The AI generates a self-contained Python 3 script (stdlib only) that reads the input file, applies the transformation, and writes the output. The script is stored.
3. Every run executes the stored script locally via python3. Runs make no model call.
4. The transformed data replaces the original and continues through the pipeline to destinations.

​

When the script is generated

• At save — Saving a pipeline with an AI transformation (POST /api/v1/pipeline, the MCP create_pipeline tool, or the UI wizard) generates the script from the source schema: the delimiter and the ordered field names and types. No sample rows are available when a pipeline is saved, so state any format the transformation depends on in the instruction (for example “dates are YYYY-MM-DD”). The script is stored in the built-in object store under pipeline-scripts/<pipeline>/, or committed to the code repository when one is enabled (see Pipeline scripts).
• Every run — Each run executes the stored script. A run makes no model call, gives the same result for the same data, and keeps working if the model is unreachable. The status shows “Running stored CodeGen script generated at …”.
• Re-saving — Saving again regenerates only when the instruction or the source schema (delimiter, field names, types or order) changed. Re-saving an unchanged pipeline makes no model call. A changed CodeGen model does not regenerate: the script keeps the model that wrote it.
• Pending — If the script cannot be generated at save (no AI key, model unreachable, AI disabled), the pipeline is still saved, the save response carries a warning and codegenScripts lists the script as pending with the reason. JSON and XML pipelines have no column schema to generate from, so they are always saved pending. The first run generates the script from its data, stores it, and later runs reuse it.
• Header differs — If the header that reaches the transformation differs from the one the script was written for (for example a column was removed by field protection or is missing from the file), that run generates a script for itself only and keeps the stored one. The status says why.
• Failures — A stored script that exits with an error or times out fails the run. It is not regenerated automatically: the error names when the script was generated and the regenerate endpoint.

​

Values withheld

​

Works with all file types

• CSV/delimited files — The script reads and writes CSV with the appropriate delimiter.
• JSON files — The script parses JSON, transforms records, and writes JSON.
• XML files — The script uses xml.etree.ElementTree to parse, transform, and write XML.

​

CLI

datris ingest data.csv --dest postgres --ai-transform "convert dates to YYYY/MM/DD and uppercase all names"

​

Requirements

• ai.enabled: true must be set in application.yaml
• The Vault secret for the AI provider must be configured (see AI Configuration)
• python3 must be available where the script runs: the datris-codegen-runner container on a compose install (USE_CODEGEN_RUNNER=true, the default), or the server itself when scripts run in-process (USE_CODEGEN_RUNNER=false)