> ## Documentation Index
> Fetch the complete documentation index at: https://docs.datris.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# AI Data Profiling

> AI-powered data profiling

Upload any data file and receive an AI-generated profile — summary statistics, data quality issues, and recommendations for validation rules and transformations. Use profiling to understand your data before setting up a pipeline configuration.

## Endpoint

`POST /api/v1/pipeline/profile`

## Parameters

| Parameter | Type | Default | Description |
| - | - | - | - |
| `file` | multipart file | (required) | The data file to profile (CSV, JSON, or XML) |
| `delimiter` | string | `,` | CSV delimiter character |
| `header` | boolean | `true` | Whether the CSV file has a header row |
| `sampleSize` | int | `200` | Number of rows to sample for large files |

## Example

```bash theme={null}
curl -s -X POST http://localhost:8080/api/v1/pipeline/profile \
  -F "file=@stock_prices.csv" \
  -F "delimiter=," \
  -F "header=true" \
  -F "sampleSize=200" | python3 -m json.tool
```

## Response

The endpoint returns a JSON object with three sections:

```json theme={null}
{
  "summary": {
    "rowCount": 200,
    "columnCount": 8,
    "columns": [
      {
        "name": "symbol",
        "inferredType": "string",
        "nullCount": 0,
        "uniqueCount": 195,
        "sampleValues": ["FAX", "IAF", "FCO"]
      },
      {
        "name": "date",
        "inferredType": "date",
        "nullCount": 0,
        "uniqueCount": 1,
        "sampleValues": ["2016-12-30"]
      },
      {
        "name": "open",
        "inferredType": "float",
        "nullCount": 1,
        "uniqueCount": 198,
        "sampleValues": ["4.65", "5.44", "7.91"]
      }
    ]
  },
  "qualityIssues": [
    "Column 'open' has 1 null/empty value",
    "Column 'volume' contains a negative value (-500)",
    "Column 'close' has a value exceeding $1,000,000"
  ],
  "recommendations": [
    "Add an aiRule to validate symbol format, price ranges, and cross-column relationships",
    "Consider sampling for large files to keep processing fast"
  ],
  "suggestedDataQuality": {
    "aiRule": {
      "instruction": "symbol must be 1-5 uppercase letters, all price columns (open, high, low, close, adj_close) must be positive and not exceed $1,000,000, volume must be positive, and high must be greater than or equal to low",
      "onFailureIsError": false
    }
  }
}
```

### Response fields

| Section | Description |
| - | - |
| `summary` | Row count, column count, and per-column statistics (inferred type, null count, unique count, sample values) |
| `qualityIssues` | Data quality problems detected in the sample — missing values, outliers, inconsistent formats, suspicious patterns |
| `recommendations` | Human-readable suggestions for validation rules and transformations |
| `suggestedDataQuality` | Ready-to-use `dataQuality` JSON block that can be copied directly into a pipeline configuration. Contains an `aiRule` with a comprehensive plain-English instruction covering all validation checks |

### Suggested data quality rules

The `suggestedDataQuality` section provides a complete, copy-paste-ready `dataQuality` configuration based on what the AI observed in the data:

* **`aiRule`** — a single comprehensive plain-English instruction covering all validation checks: format patterns (emails, phone numbers, dates), value ranges, cross-column relationships (e.g., high >= low), and business logic. If no validation rule is appropriate, this field is omitted.

See [AI Rule](/data-quality/ai-rules) for full documentation.

## How it works

1. You upload a file — no pipeline registration needed
2. For large CSV files, the pipeline randomly samples `sampleSize` rows (keeping the header)
3. The sampled content is sent to the AI model with a profiling prompt
4. The AI analyzes the data and returns a structured JSON profile

Profiling is a standalone operation — it does not require a registered pipeline, data quality rules, or any pipeline configuration. It is designed to be the first step when working with a new data source, before setting up validation or ingestion.

## Use cases

* **Explore new data** — understand the structure, types, and quality of an unfamiliar file before writing a pipeline configuration
* **Discover quality issues** — find missing values, outliers, format inconsistencies, and suspicious patterns
* **Generate rule ideas** — the AI suggests aiRule instructions and transformations based on what it observes
* **Validate assumptions** — confirm that a file matches expected schema and data quality before loading

## Sampling

For CSV files larger than `sampleSize` rows, the profiling endpoint automatically samples a random subset. The header row is always included. For JSON and XML files, the content is truncated if it exceeds the AI model's context window.

The default sample of 200 rows is typically sufficient to detect patterns, types, and quality issues. Increase `sampleSize` for more thorough profiling at the cost of slower response times.

## When the model declines

An AI provider's safety classifier can decline to look at a file, most often one that looks like personal data. Profiling then still returns `200` with the server's own statistics only: `summary.rowCount`, `summary.columnCount` and `summary.columns` computed over every row of a CSV file (inferred type, nulls, distinct count, value lengths; `sampleValues` empty; empty for JSON and XML files), empty `qualityIssues` and `recommendations`, no `suggestedDataQuality`, a top-level `"aiDeclined": true` and a fixed `note`: "the model declined this request; statistics only". The provider's own text is never included. Read an empty `qualityIssues` list with `aiDeclined` as "no AI review", not as a clean file.

| Provider family | Recognised as a decline |
| - | - |
| Anthropic (direct or AWS Bedrock) | `stop_reason: "refusal"` |
| OpenAI, Azure OpenAI, Grok and Ollama (chat completions, streamed or not) | `finish_reason: "content_filter"`, or a non-empty `message.refusal` (`delta.refusal` when streamed) |
| OpenAI Responses API models | an output item or message content part of type `refusal`, or `incomplete_details.reason: "content_filter"` |

Only response-side declines are recognised. Azure OpenAI's prompt-side content filter answers with HTTP 400 (`code: content_filter`) and is still returned as an error. Nothing else counts as a decline: a missing or wrong provider key, a timeout, a rate limit or any other provider error is still returned as an error, so a misconfiguration is not hidden.

Other AI features (search answers, CodeGen, tap generation, fix suggestions and the rest) have no local fallback: a decline there, including a `content_filter` finish that used to return the cut-off text, is reported as an error.

## Values withheld

When the server sets `DATRIS_AI_SAMPLE_VALUES=false`, no row value is sent to the model. For a CSV file the model gets per-column statistics computed over every row (inferred type, count, nulls, distinct count, minimum and maximum value length); for JSON, NDJSON and XML (by the `.json`, `.ndjson`, `.jsonl` or `.xml` extension; for an unrecognised extension, by content that starts with `{`, `[` or `<`) it gets a value-free skeleton of the document. `.csv`, `.tsv`, `.psv` and `.txt` files are always profiled as delimited. A header cell that does not look like a plain column name is sent as `column_N`, and a first line that looks like data (a dash next to a digit, an `@`, or a long run of digits in any cell) is counted as a row and every column is numbered. A cell with more than two digits or no letter at all also marks line 1 as data. A headerless first row made only of words still passes as names and is sent: single words in any case (`John,Smith,Boston,MA`; `jane,doe,asthma,f`; `Diabetes,Type,Two`), all-caps or lower-case multi-word cells (`JOHN SMITH`, `jane doe`), and names the capitalised-words test misses (`John A Smith`, `McDonald Smith`, `O Brien`). `header` defaults to `true`: profile a headerless file with `header=false` so no cell of its first line is sent. `sampleValues` are empty for every column and the response carries a top-level `"valuesWithheld": true`. Quality issues and the suggested rule are then based on names, types and statistics only. If the model's reply cannot be read as JSON (for example a very wide file whose reply is cut off), or the model call fails for any reason (the model declines the request, a provider error, a timeout), the response holds the server's statistics only: `summary.columns` (empty for JSON and XML files), empty `qualityIssues` and `recommendations`, `"valuesWithheld": true` and a `note` giving the cause in the server's own words; the provider's error text is never included. A missing or misconfigured provider key or model also produces this note when values are withheld; check it with [`datris doctor`](/doctor). With values sent (the default), a failed model call still returns an error; only a model decline falls back, as described in [When the model declines](#when-the-model-declines) (with `aiDeclined`, not `valuesWithheld`). See [Field Protection](/transformation/field-protection#what-still-sees-raw-data).

## Requirements

* `ai.enabled: true` must be set in `application.yaml`
* A configured AI provider (see [AI Configuration](/ai-configuration))
* Cloud providers (Anthropic, OpenAI, Azure OpenAI, Amazon Bedrock, Grok) recommended for best results


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.