AI Data Profiling

Upload any data file and receive an AI-generated profile — summary statistics, data quality issues, and recommendations for validation rules and transformations. Use profiling to understand your data before setting up a pipeline configuration.

Endpoint

POST /api/v1/pipeline/profile

Parameters

Parameter	Type	Default	Description
`file`	multipart file	(required)	The data file to profile (CSV, JSON, or XML)
`delimiter`	string	`,`	CSV delimiter character
`header`	boolean	`true`	Whether the CSV file has a header row
`sampleSize`	int	`200`	Number of rows to sample for large files

Example

curl -s -X POST http://localhost:8080/api/v1/pipeline/profile \
  -F "file=@stock_prices.csv" \
  -F "delimiter=," \
  -F "header=true" \
  -F "sampleSize=200" | python3 -m json.tool

Response

The endpoint returns a JSON object with three sections:

{
  "summary": {
    "rowCount": 200,
    "columnCount": 8,
    "columns": [
      {
        "name": "symbol",
        "inferredType": "string",
        "nullCount": 0,
        "uniqueCount": 195,
        "sampleValues": ["FAX", "IAF", "FCO"]
      },
      {
        "name": "date",
        "inferredType": "date",
        "nullCount": 0,
        "uniqueCount": 1,
        "sampleValues": ["2016-12-30"]
      },
      {
        "name": "open",
        "inferredType": "float",
        "nullCount": 1,
        "uniqueCount": 198,
        "sampleValues": ["4.65", "5.44", "7.91"]
      }
    ]
  },
  "qualityIssues": [
    "Column 'open' has 1 null/empty value",
    "Column 'volume' contains a negative value (-500)",
    "Column 'close' has a value exceeding $1,000,000"
  ],
  "recommendations": [
    "Add an aiRule to validate symbol format, price ranges, and cross-column relationships",
    "Consider sampling for large files to keep processing fast"
  ],
  "suggestedDataQuality": {
    "aiRule": {
      "instruction": "symbol must be 1-5 uppercase letters, all price columns (open, high, low, close, adj_close) must be positive and not exceed $1,000,000, volume must be positive, and high must be greater than or equal to low",
      "onFailureIsError": false
    }
  }
}

Response fields

Section	Description
`summary`	Row count, column count, and per-column statistics (inferred type, null count, unique count, sample values)
`qualityIssues`	Data quality problems detected in the sample — missing values, outliers, inconsistent formats, suspicious patterns
`recommendations`	Human-readable suggestions for validation rules and transformations
`suggestedDataQuality`	Ready-to-use `dataQuality` JSON block that can be copied directly into a pipeline configuration. Contains an `aiRule` with a comprehensive plain-English instruction covering all validation checks

Suggested data quality rules

The suggestedDataQuality section provides a complete, copy-paste-ready dataQuality configuration based on what the AI observed in the data:

aiRule — a single comprehensive plain-English instruction covering all validation checks: format patterns (emails, phone numbers, dates), value ranges, cross-column relationships (e.g., high >= low), and business logic. If no validation rule is appropriate, this field is omitted.

See AI Rule for full documentation.

How it works

You upload a file — no pipeline registration needed
For large CSV files, the pipeline randomly samples sampleSize rows (keeping the header)
The sampled content is sent to the AI model with a profiling prompt
The AI analyzes the data and returns a structured JSON profile

Profiling is a standalone operation — it does not require a registered pipeline, data quality rules, or any pipeline configuration. It is designed to be the first step when working with a new data source, before setting up validation or ingestion.

Use cases

Explore new data — understand the structure, types, and quality of an unfamiliar file before writing a pipeline configuration
Discover quality issues — find missing values, outliers, format inconsistencies, and suspicious patterns
Generate rule ideas — the AI suggests aiRule instructions and transformations based on what it observes
Validate assumptions — confirm that a file matches expected schema and data quality before loading

Sampling

For CSV files larger than sampleSize rows, the profiling endpoint automatically samples a random subset. The header row is always included. For JSON and XML files, the content is truncated if it exceeds the AI model’s context window. The default sample of 200 rows is typically sufficient to detect patterns, types, and quality issues. Increase sampleSize for more thorough profiling at the cost of slower response times.

When the model declines

An AI provider’s safety classifier can decline to look at a file, most often one that looks like personal data. Profiling then still returns 200 with the server’s own statistics only: summary.rowCount, summary.columnCount and summary.columns computed over every row of a CSV file (inferred type, nulls, distinct count, value lengths; sampleValues empty; empty for JSON and XML files), empty qualityIssues and recommendations, no suggestedDataQuality, a top-level "aiDeclined": true and a fixed note: “the model declined this request; statistics only”. The provider’s own text is never included. Read an empty qualityIssues list with aiDeclined as “no AI review”, not as a clean file.

Provider family	Recognised as a decline
Anthropic (direct or AWS Bedrock)	`stop_reason: "refusal"`
OpenAI, Azure OpenAI, Grok and Ollama (chat completions, streamed or not)	`finish_reason: "content_filter"`, or a non-empty `message.refusal` (`delta.refusal` when streamed)
OpenAI Responses API models	an output item or message content part of type `refusal`, or `incomplete_details.reason: "content_filter"`

Only response-side declines are recognised. Azure OpenAI’s prompt-side content filter answers with HTTP 400 (code: content_filter) and is still returned as an error. Nothing else counts as a decline: a missing or wrong provider key, a timeout, a rate limit or any other provider error is still returned as an error, so a misconfiguration is not hidden. Other AI features (search answers, CodeGen, tap generation, fix suggestions and the rest) have no local fallback: a decline there, including a content_filter finish that used to return the cut-off text, is reported as an error.

Values withheld

When the server sets DATRIS_AI_SAMPLE_VALUES=false, no row value is sent to the model. For a CSV file the model gets per-column statistics computed over every row (inferred type, count, nulls, distinct count, minimum and maximum value length); for JSON, NDJSON and XML (by the .json, .ndjson, .jsonl or .xml extension; for an unrecognised extension, by content that starts with {, [ or <) it gets a value-free skeleton of the document. .csv, .tsv, .psv and .txt files are always profiled as delimited. A header cell that does not look like a plain column name is sent as column_N, and a first line that looks like data (a dash next to a digit, an @, or a long run of digits in any cell) is counted as a row and every column is numbered. A cell with more than two digits or no letter at all also marks line 1 as data. A headerless first row made only of words still passes as names and is sent: single words in any case (John,Smith,Boston,MA; jane,doe,asthma,f; Diabetes,Type,Two), all-caps or lower-case multi-word cells (JOHN SMITH, jane doe), and names the capitalised-words test misses (John A Smith, McDonald Smith, O Brien). header defaults to true: profile a headerless file with header=false so no cell of its first line is sent. sampleValues are empty for every column and the response carries a top-level "valuesWithheld": true. Quality issues and the suggested rule are then based on names, types and statistics only. If the model’s reply cannot be read as JSON (for example a very wide file whose reply is cut off), or the model call fails for any reason (the model declines the request, a provider error, a timeout), the response holds the server’s statistics only: summary.columns (empty for JSON and XML files), empty qualityIssues and recommendations, "valuesWithheld": true and a note giving the cause in the server’s own words; the provider’s error text is never included. A missing or misconfigured provider key or model also produces this note when values are withheld; check it with datris doctor. With values sent (the default), a failed model call still returns an error; only a model decline falls back, as described in When the model declines (with aiDeclined, not valuesWithheld). See Field Protection.

Requirements

ai.enabled: true must be set in application.yaml
A configured AI provider (see AI Configuration)
Cloud providers (Anthropic, OpenAI, Azure OpenAI, Amazon Bedrock, Grok) recommended for best results

Getting Started

Connect your agent

Assistant

Catalog

Taps

Ingestion

Destinations

Data Quality

Transformation

AI Features

Configuration

Governance & Security

Production

Integrations

Recipes

Examples

AI Data Profiling

Endpoint

Parameters

Example

Response

Response fields

Suggested data quality rules

How it works

Use cases

Sampling

When the model declines

Values withheld

Requirements

​Endpoint

​Parameters

​Example

​Response

​Response fields

​Suggested data quality rules

​How it works

​Use cases

​Sampling

​When the model declines

​Values withheld

​Requirements

Endpoint

Parameters

Example

Response

Response fields

Suggested data quality rules

How it works

Use cases

Sampling

When the model declines

Values withheld

Requirements