Data Quality — AI Rule

​

Configuration

​

How it works

1. Prompt generation — At pipeline save, Datris combines the source schema (delimiter, column names and types) with your instruction and sends a single prompt to the configured AI model. When a run has to generate the script instead (see When the script is generated), it uses the column names and a few sample rows from that run’s data.
2. Script generation — The AI model generates a self-contained Python 3 validation script (stdlib only, no pip packages).
3. Local execution — On every run, the stored script is written to a temp file and executed via python3 against the full data file. Runs make no model call.
4. Result parsing — The script outputs a JSON array of failures: [{"index": <row_number>, "reason": "..."}]. Datris parses this and maps failures back to the original data. The index is 0-based: the first record is index 0, the second is index 1, and so on.
5. Cleanup — Temp files are deleted after execution.

​

Example

{
  "dataQuality": {
    "aiRule": {
      "instruction": "Validate that all email addresses are properly formatted, all phone numbers contain 7-15 digits, all dates are in YYYY-MM-DD format, and prices are positive and not exceeding $1,000,000",
      "onFailureIsError": true
    }
  }
}

​

Works with all file types

• CSV/delimited files — The script reads the CSV with the appropriate delimiter and validates each row.
• JSON files — The script parses the JSON array and validates each record.
• XML files — The script uses xml.etree.ElementTree to parse and validate each element.

​

When the script is generated

• At save — Saving a pipeline with an AI rule (POST /api/v1/pipeline, the MCP create_pipeline tool, or the UI wizard) generates the script from the source schema: the delimiter and the ordered field names and types. No sample rows are available when a pipeline is saved, so state any format the rule depends on in the instruction (for example “dates are YYYY-MM-DD”). The script is stored in the built-in object store under pipeline-scripts/<pipeline>/, or committed to the code repository when one is enabled (see Pipeline scripts).
• Every run — Each run executes the stored script. A run makes no model call, gives the same result for the same data, and keeps working if the model is unreachable. The status shows “Running stored CodeGen script generated at …”.
• Re-saving — Saving again regenerates only when the instruction or the source schema (delimiter, field names, types or order) changed. Re-saving an unchanged pipeline makes no model call. A changed CodeGen model does not regenerate: the script keeps the model that wrote it.
• Pending — If the script cannot be generated at save (no AI key, model unreachable, AI disabled), the pipeline is still saved, the save response carries a warning and codegenScripts lists the script as pending with the reason. JSON and XML pipelines have no column schema to generate from, so they are always saved pending. The first run generates the script from its data, stores it, and later runs reuse it.
• Header differs — If the header that reaches the rule differs from the one the script was written for (for example a column was removed by field protection or is missing from the file), that run generates a script for itself only and keeps the stored one. The status says why.
• Failures — A stored script that exits with an error, times out, or prints output that is not a JSON array fails the run. It is not regenerated automatically: the error names when the script was generated and the regenerate endpoint.

​

Values withheld

​

Behavior

• An aiRule whose instruction is empty or only whitespace is ignored: no script is generated at save and the run skips the stage with a status line saying so.
• The AI rule’s Python script is generated when the pipeline is saved (or by the first run that needs one) and reused by every run until the instruction or source schema changes
• The generated script runs via python3 with a 5-minute timeout, in the isolated datris-codegen-runner container on a compose install (see below)
• Scripts are constrained to Python standard library only (no external packages)
• Rules with onFailureIsError: true are counted as errors; onFailureIsError: false are warnings
• All rows are evaluated; the abort message lists the first 100 errors and a count of any remaining
• After all rows are evaluated, if any errors exist, processing is aborted
• Warnings are summarized at the end of validation

​

Where the script runs

​

Requirements

• ai.enabled: true must be set in application.yaml
• The Vault secret for the AI provider must be configured (see AI Configuration)