Skip to main content
Field protection rewrites sensitive columns as the very first step after ingest, so every later stage only ever sees the protected value. You opt in per field with a protect object on the source schema.

​
Why

Datris sends samples of your data to an AI provider when a pipeline uses an AI rule, an AI transformation, or error explanation. Field protection runs before all of them. A protected column reaches the AI provider, Live Read, and every destination only in its protected form, so the raw value of that column is never sent to a model provider. That means you do not need a business associate agreement (or a similar data-processing agreement) with the provider for the columns you protect. Columns you do not protect are sent as they are. Protection is local: no value leaves the Datris server to be protected, and no status line, error message, or log line carries a raw value.

​
Methods

Empty values stay empty for every method. An encrypt value is longer than its input: enc:v<n>: followed by the base64url of the input’s UTF-8 bytes plus 28 bytes (a 12-byte IV and a 16-byte authentication tag). A 20-character email becomes 71 characters. If the destination column is a varchar(n), size it for the ciphertext, not the original value.

​
Config example

To keep the ingest object after protection, add "protection": {"purgeSource": false} at the top level (see Source retention). From an agent, create_pipeline takes the same policies as a protect map keyed by field name, for example {"mrn": {"method": "hmac"}, "ssn": {"method": "drop"}}.

​
Suggestions

Datris can propose a starting point. The configured CodeGen model receives the field names and types only (one name:type line per field) and returns, for each field, hmac, mask with an optional preserve, redact, drop, or no protection (it never suggests encrypt; choose that yourself), with a one-line reason. No row value is sent, and nothing is saved: you confirm the fields you want and then save them as protect.
Pass {"pipeline": "<name>"} to use a saved pipeline’s source schema, or {"fields": [{"name": "...", "type": "..."}]} for a schema you have not saved yet. Each entry in the response has name, type, suggested ({"method", "preserve"}, or null for no protection), reason, and current: the field’s existing protect, so running it again on a protected pipeline never reads as an overwrite. The model field names the model that answered. Every suggestion follows the type rules: a field that is not string is only ever suggested drop or nothing, and preserve only comes with mask. With pipeline, the saved destination is checked too: a keyFields column is only ever suggested hmac or nothing, and a field whose destination type is not string is only suggested drop or nothing. A JSON pipeline’s _json field is not sent to the model; list its top-level keys in fields instead. One call takes at most 150 fields; for a wider table, pass subsets through fields (more returns HTTP 400 before the model is called). The call needs the same permission as reading a pipeline, is recorded as a pipeline / protect-suggest event in the audit log (field names only, failed model calls included), and returns HTTP 400 with {"error": "..."} when AI is not configured. Names can mislead, so treat the answer as a proposal. From an agent, the suggest_field_protection tool returns one line per field; the agent shows them to you and asks which to accept before passing any of them as protect to create_pipeline.

​
HIPAA Safe Harbor preset

For health data, set "protection": {"preset": "hipaa-safe-harbor"} at the top level of the pipeline instead of classifying every column by hand. The preset recognises the 18 identifier classes of the HIPAA Safe Harbor method (45 CFR 164.514(b)(2)) from field names, using a fixed table built into the server. No model is called, so the same names always give the same answer.
The preset is an aid to applying Safe Harbor, not a certification of compliance. It protects the fields it recognises by name and tells you what it could not decide. Whether a data set is de-identified is still your determination. Expert Determination and business associate agreements are out of scope.

​
Classes and default methods

Before a name is looked up it is lowercased, split on separators and camelCase, and a leading word such as patient, pt, member, subscriber or guarantor is dropped, so patient_email, memberEmail and email are the same field. The rest of the name must match one of the table’s names exactly, so name_of_drug, zip_file, phone_model and date_format are not classified. A trailing number is ignored (phone2 is a phone). Stable identifiers use hmac so the column still joins. Phone and fax numbers are redacted rather than masked to the last four digits, because a partial number is not Safe Harbor. The name is looked up with its leading prefixes dropped one at a time, longest first, so member_id, patient_member_id and subscriber_number_2 are health plan numbers. A bare id (including patient_id once the prefix is dropped) is not classified, so key columns are not swept up; other_id needs an explicit word such as national_id or unique_identifier. Ages are left unclassified, as are free-text columns. Person-name variants such as username, display_name, doctor_name, provider_name, first and last are not classified either; protect them yourself if they name a patient or a relative. Generic and event dates (date, created_at, updated_at, timestamp, start_date, order_date, due_date) are deliberately left unclassified so the preset does not destroy operational timestamps; the proposal lists them in a review note.

​
What the preset does to common column names

The defaults are chosen for health data. If you opt in on another kind of data set, expect these effects on ordinary-looking columns:
  • Dropped (the column is removed): image, picture, image_url, photo, photo_url, profile_image, lat, lng, lon, latitude, longitude, signature, ssn and its variants.
  • Redacted ([REDACTED]): a bare name, city, address, cell, mobile, phone, email, url, website, ip.
  • Pseudonymised (hmac, still joins): account, acct, license, plate, serial_number, member_id, mrn, card_number.
  • Year only: dob, dos, admission, discharge, visit and appointment dates.
At save these show up as refusals you can answer with protect or presetExempt. Schema evolution is different: a new column in one of these classes is protected, or removed for a drop class, in the same run, and the run’s status line is the only trace. Exempt such a name in advance if you need it in the clear.

​
Proposal endpoint

Pass {"preset", "pipeline": "<name>"} to use a saved pipeline’s source schema, or {"preset", "fields": [...]}. The response has fields (each recognised field with name, class, method, preserve, reason, and current, the field’s existing protect or null), unclassified (the names the table did not recognise, for a person to look at), and review (notes for rules a method cannot express). Each proposed method follows the type rules the same way suggestions do: a keyFields column is only proposed hmac or nothing, and a field whose source or destination type is not string is only proposed drop or nothing; method is "none" in that case and reason says why. _json and _xml are skipped, and one call takes at most 150 fields. With pipeline, a JSON, XML or unstructured pipeline is refused with HTTP 400 rather than answered with an empty proposal, because the preset cannot see its keys; pass the keys to check as fields instead. The call needs the same permission as reading a pipeline. It is stateless, saves nothing and is not recorded in the audit log, because it reads no data and sends nothing out. For fields the table does not recognise you can still ask the suggestions endpoint; the preset never calls it.

​
Review notes

The proposal adds a note instead of pretending a method covers these rules:
  • Ages over 89. Safe Harbor requires ages over 89 (and the dates or years that reveal them) to be aggregated into a single category of 90 or older. The preset does not aggregate ages; the note appears when an age or date field is present.
  • Small-population ZIP prefixes. The first three ZIP digits must be replaced with 000 where the area they cover holds 20,000 people or fewer. first3 keeps the first three digits and does not check population; the note appears when a ZIP field is present.
  • Free text. Columns such as notes, comments or description may contain identifiers inside their values. The preset classifies by name only; the note names those columns.
  • Other dates. Unclassified columns whose name ends in date, dt, at, time or timestamp (created_at, start_date, event_time) may be dates related to an individual. The preset did not mask them; the note names them so you can protect the ones that belong to a patient.

​
Enforcement

When a pipeline sets the preset:
  • At save, a field the table recognises that has no protect is refused, for example Preset 'hipaa-safe-harbor': field 'phone' looks like phone and has no protection. Add protect to it or list it under protection.presetExempt. This applies even when no field has protect yet. When the field’s source or destination type is not string, only drop can protect it, and the message says so. An unknown preset name is refused too.
  • Delimited sources only. The preset is refused on a JSON pipeline, because a JSON pipeline’s keys are not schema fields the preset could check or evolve: list the keys to protect with protect explicitly (the proposal endpoint accepts any field list), and protect any key that appears later yourself. It is refused on XML and unstructured sources, like protect.
  • For new columns, a column added later by schema evolution that the table recognises is added with its default method (clamped by the type rules) and protected in that same run, before any later stage or destination sees it. The run status says, for example, Preset hipaa-safe-harbor: new column 'patient_phone' protected as phone (redact), and the stored pipeline shows the new field’s protect. A new column the table does not recognise is added unprotected, as on any pipeline, with a warning: new column '<name>' is not recognised by the preset and is not protected. A new column listed under presetExempt is added unprotected with an info line. A recognised new column whose default method the type rules refuse (for example a keyFields column given anything but hmac) lands as is with a warning, and the next save of the pipeline is refused until you protect it or exempt it.
To keep a recognised field in the clear on purpose, list it under presetExempt:
Without the preset, a column added by schema evolution to a protected pipeline lands unprotected; that is the main reason to set it. The pipeline wizard applies the preset without writing JSON; see In the wizard. An agent does the same through MCP with suggest_field_protection (preset) and create_pipeline (protect_preset, protect_exempt).

​
In the wizard

In the pipeline wizard’s Source Schema step, every field has a Protect select (None, hmac, mask, redact, drop, encrypt), and choosing mask adds a Keep select (mask all, last 4, email domain, year, first 3). A choice the type rules do not allow, such as hmac on an int field, switches back to None with the reason shown under the field. A column you chose as a key field is offered only None and hmac (a suggestion for any other method is declined with the reason), and a field protected with anything but hmac is left out of the destination’s Key Fields list. Suggest protection sends the field names and types only, never data, to the suggestions endpoint and fills each empty Protect select with its suggestion, marked with the reason until you Keep it (which applies the suggestion, even over a value you had already chosen) or Clear it; nothing is saved until you save the pipeline. JSON and XML pipelines show no Protect column, so set protect on a JSON pipeline’s top-level keys in the config JSON instead. The pipeline’s detail page lists its protected fields and their methods. On a delimited (CSV) source, the Preset select beside Suggest protection offers HIPAA Safe Harbor. Choose it and click Apply: the field names and types go to the proposal endpoint, every recognised field you have not set yet gets the preset’s method, and each recognised field shows its class beside the Protect select. A field you had already set keeps your value and shows what the preset proposes, with Keep to take it or Dismiss to leave yours; a proposal the type rules refuse is shown as not applied with the reason. Below the fields, the wizard lists the fields the preset did not recognise and the review notes; read both, because the preset is an aid, not a compliance determination. Keep the preset enforced on this pipeline is ticked by default and saves protection.preset, so later saves and new columns stay covered (see Enforcement); untick it to keep the methods without the preset. While the preset is enforced, a recognised field set to None shows an Exempt toggle, which adds it to presetExempt. The summary lists any recognised field that is None and not exempt, because the save would be refused, and an exemption for a field no longer in the schema is dropped on save. Applying the preset again leaves exempt fields at None. Reopening the pipeline restores the preset, the exemptions and the class labels without changing the saved methods, and the detail page shows Preset: HIPAA Safe Harbor (enforced) with any exempt fields.

​
Where it runs

The stage runs once per job, after the preprocessor and before data quality, transformation, Live Read, and every destination. The preprocessor still sees raw values; if your preprocessor endpoint must not receive them, protect the data upstream or do not use a preprocessor on that pipeline. Each run that protects data adds one status line such as Protected 3 fields: mrn=hmac, email=mask:domain, ssn=drop. Column-level lineage shows protected fields as derived with the method as evidence, and dropped fields as dropped.

​
Type rules

  • hmac, mask, redact, and encrypt produce a string, so the field must be string in the source schema and, when a destination schema lists it, string there too.
  • drop works on a field of any type.
  • A column named in the destination’s keyFields may only use hmac: mask and redact would give different rows the same key (so upserts would merge them), encrypt gives the same value a different ciphertext on every run (so upserts would never match), and drop would remove the key.
  • The source must be delimited (CSV and similar) or JSON. For JSON pipelines the source schema lists _json plus one entry for each protected top-level key; nested keys are not supported yet.
  • Unknown methods, preserve on anything but mask, and the reserved methods fpe and tokenize are rejected when the pipeline is saved. Like other validation errors, the save returns HTTP 400 with {"error": "..."} naming the field.

​
The key

hmac uses one key per environment, stored in Vault as the secret <environment>/field-protection, field key. The server issues it the first time a protected run needs it and records an key / issue / field-protection event in the audit log. The key is read fail-closed: if Vault cannot be read, the run fails rather than writing data with a new key. Datris never rotates the key automatically. Rotating it yourself changes every pseudonym, so new values will no longer match or join with data already landed. encrypt uses separate, versioned keys in the same secret: fields enc.v1, enc.v2, and so on, with encCurrent naming the version new values are encrypted with. The server issues enc.v1 the first time an encrypt run needs it (audited as key / issue / field-protection). Keys never leave the server and no user ever holds one. See Key rotation. The field-protection secret cannot be referenced by a tap or a pipeline.

​
Revealing a value

encrypt is the one reversible method. To read an original value back, send the ciphertexts you read from the destination to the reveal endpoint:
The response has one slot per input value, in order:
  • Who can call it: an API key granted protect:reveal, or an admin session. No key template and neither the editor nor the viewer role carries it, so it must be granted explicitly in the API Keys tab (the full-access template and legacy unscoped keys hold it through *:*). There is no MCP tool, Assistant action, or CLI command for it (the only UI path is the Search screen below, which calls this same endpoint), and the recovery agent’s key does not have it, so an agent can never reveal a value. The reveal and rotate endpoints enforce their capability themselves, so CAPABILITY_ENFORCEMENT=log-only does not open them.
  • What is checked: the field must carry protect with method encrypt (otherwise HTTP 400). Each ciphertext is bound to its pipeline and field name, so a value copied to another pipeline or column does not reveal: that slot is null and errors gets {"index", "message"}, and the rest of the call still succeeds.
  • Limit: at most 1000 values per call (more returns HTTP 400).
  • What is logged: every call is a protect / reveal event in the audit log naming who called, the pipeline, the field, and how many values were requested, revealed, and failed (outcome warning when any failed). The values themselves and the ciphertexts are never logged or recorded. A call without the capability is refused with HTTP 403 and recorded as a security denied event.
Reveal is stateless: Datris never queries a destination to find the values, so you choose exactly which rows to reveal. From the Search screen. In Search → Traditional, a cell holding a ciphertext shows a lock badge with its key version (enc:v2) instead of the base64. When the result set has encrypted cells, admins (or anyone, when the install uses API keys without user logins) see Reveal encrypted values in the results header; editors and viewers see the badges but no button. One click reveals every encrypted cell on screen through the endpoint above, one call per encrypted column (split at 1000 values), so the audit log gets one protect / reveal entry per column per click. Revealed cells show the plaintext with an open-lock badge; a value the server could not reveal keeps its lock and shows the server’s reason on hover. Databricks, Snowflake, and object-store queries use the pipeline selected in the form. For PostgreSQL and MongoDB the pipeline is found from the table or collection you queried; if no pipeline or more than one writes it, pick the pipeline beside the button. Without the capability the screen shows “You do not have the protect:reveal capability” and nothing is retried. Nothing is stored: plaintext stays only in the page, and running the query again shows the ciphertext.

​
Key rotation

To start encrypting with a new key, rotate it with a key that has protect:admin (or an admin session):
The response is {"version": 2}. Runs after this write enc:v2: values. The old versions stay in the secret, so rows already landed with enc:v1: still reveal; Datris does not re-encrypt landed rows. The rotation is recorded as a key / rotate / field-protection event. The hmac key is never rotated by this endpoint. To retire an old version, save the field-protection secret (Secrets tab or PUT /api/v1/secrets/field-protection) without its enc.v<n> field, using a key that has protect:admin. Never remove the current version’s field: encCurrent must keep naming a stored version. The highest version number cannot be removed, and at least one version must remain. After that, every value encrypted with that version can no longer be revealed; re-landing those rows from the original source is the only way back. The rotate endpoint never reuses a retired version number.

​
Editing the key secret through the API

The secrets API guards <environment>/field-protection:
  • key (the hmac key) cannot be changed or removed through the API by anyone. The request fails with 409 and nothing is written. Changing it changes every hmac pseudonym, so the only supported path is a deliberate edit of the secret in Vault.
  • Adding, changing or removing an enc.v<n>, or changing encCurrent needs protect:admin. A key with only secret:write gets 403 and a security / denied audit entry; nothing is written. Each enc.v<n> value must be 64 hex characters (a 32-byte AES-256 key) and encCurrent must name a stored version; otherwise the request fails with 400. The next run after an accepted edit encrypts with the new current version; no restart is needed.
  • Accepted edits are audited with the changed field names only, never the values: key / rotate / field-protection when an enc.v<n> was added or changed, key / retire / field-protection when one was removed, and key / set-current / field-protection when only encCurrent moved.
  • A save that leaves every key field as it is (fields sent back masked or blank) works with plain secret:write, so the Secrets tab can round-trip the secret.
  • Deleting the secret (DELETE /api/v1/secrets/field-protection) is refused with the same 409 for everyone: the next protected run would issue a new hmac key and every landed ciphertext would be orphaned.
  • Creating the secret through the API is refused with 409: the server issues it on the first protected run.
  • Field names are limited to key, encCurrent, enc.v<n> (with n from 1 to 1000000, no sign or leading zero) and createdByKeyLabel. Any other field, including look-alikes such as Key or enc.v05 and a _type tag, fails with 400 for every caller. encCurrent follows the same number rule.
  • Version numbers are never reused. Retiring removes older versions only: removing every enc.v<n>, or the highest version number, fails with 400. A new enc.v<n> must be numbered above the highest stored version; re-adding a retired lower number fails with 400.

​
Source retention

Protecting the loaded data is not enough if the raw file stays on disk or in a bucket. As soon as the protected copy exists, Datris deletes:
  • the raw staged file, including the payload a preprocessor received as input;
  • the ingest object, when the run came from a bucket drop, a Kafka temp object, or an archive drop (the archive and its extracted files both go).
POST /pipeline/upload streams the file straight into staging and writes no bucket object, so for uploads only the staged file applies. Each purge is recorded as a pipeline / purge-source event in the audit log. A purge that fails is a warning on the run, never a failed run. Because the raw source is gone, a later failure in the same run cannot be replayed from Datris: re-upload the file or re-run the tap. To keep the ingest object, set "protection": {"purgeSource": false} at the top level of the pipeline. The staged file is still deleted. Pipelines without any protect field never purge anything.

​
What still sees raw data

Field protection applies inside a pipeline run. A few helpers that write configuration or code read your data before protection applies, or read columns you have not protected:
  • schema generation from an uploaded file (POST /pipeline/generate, the wizard’s Generate Schema and sample-file analysis);
  • data profiling (POST /pipeline/profile and the profile_data MCP tool);
  • JSON Schema and XSD generation from a pasted sample (POST /config/generate-schema);
  • files attached to an Assistant chat;
  • the sample rows the CodeGen data-quality rule and transformation generators put in their prompt (protected columns are already protected there; the other columns are sent as they are);
  • the AI fix suggestion and recovery-agent diagnosis for a failed pipeline run, whose error text can quote row values (data-quality failure reasons, a script’s error output).
If you handle regulated data, set DATRIS_AI_SAMPLE_VALUES=false on the datris service and recreate it. With the switch off, none of these helpers sends a row value to a model. Each one works from structure only and says so in its response: What is still sent is structure, and some structure can carry data, so these rules apply:
  • Column names from a file. Schema generation, profiling and the CodeGen prompts take the header flag as a statement that line 1 is a header. With header=true, line 1 is sent as column names only when it looks like a header, cell by cell:
    • a cell is sent as a name only if it uses letters, digits, underscores and spaces (no dash, dot or other punctuation), starts with a letter or underscore, has at most 64 characters and at most two digits, is not all digits, and is not two or more capitalised words (a full name). Any other cell is sent as column_N. So full_name, email, first name, addr1 and zip_code are sent; First Name, e-mail and account123 become column_N.
    • if any cell has a dash next to a digit (MRN-12345, 2024-01-05), an @, more than four digits in a row, more than two digits, or no letter at all (42, 555 1234), the whole line is treated as data and every column is numbered.
    This check cannot recognise every value. A headerless first row made only of words still passes as names and is sent as they are: single words in any case (John,Smith,Boston,MA; jane,doe,asthma,f; Diabetes,Type,Two), all-caps or lower-case multi-word cells (JOHN SMITH, jane doe), and names the capitalised-words test misses (John A Smith, McDonald Smith, O Brien). Profile or generate a headerless file with header=false (the profiling endpoint defaults to true); then no cell of line 1 is sent. The run-time header check (dataQuality.validateFileHeader) is not covered by the switch: it sends line 1 to the model as it is.
  • Assistant attachments have no header option, so line 1 of a CSV attachment is never sent: columns are listed as column_1, column_2, … and every line counts as a row.
  • Stored schema field names. The field names saved in a pipeline’s schema are treated as operator-entered and are always sent to the CodeGen generators. With the switch off, the ai.sample_values doctor check warns, naming up to five pipelines, when a stored source field name does not read as a column name.
  • JSON keys. Key names are sent, except where an object reads as a map keyed by data: a key that has a dash next to a digit, an @, more than four digits in a row or more than two digits, that is two or more capitalised words, or that uses punctuation other than _, ., - and space; more than 50 keys; two or more keys whose values are all objects of the same shape (records keyed by a person’s name or an id); or, below the top level, two or more keys whose values are all arrays ({"patients": {"smith": [..], "jones": [..]}}). Such an object is sent as {"<key>": <value shape>}. Otherwise keys that are plain words, including john.smith or jane-doe, are sent: a map keyed by names is collapsed only when its values are objects of one shape (or, below the top level, all arrays). A map keyed by plain names whose values are strings, numbers or booleans is sent with its keys, at the top level ({"smith": "a", "jones": "b"}) or nested ({"scores": {"smith": 1, "jones": 2}}), because a record’s own fields ("address": {"city": .., "zip": ..}) have exactly that shape. Keys such as zip-code or content.type are kept.
  • XML. Element and attribute names are sent. Text, attribute values and namespace URIs are not (namespace prefixes are kept).
  • Unparseable files are described only by their type and size, never by their content.
The default is true (today’s behaviour). Any value other than false counts as true and is logged once, so set false explicitly. The ai.sample_values doctor check reports the effective value and warns when a pipeline protects fields while the switch is on. The switch does not cover features whose purpose is to answer questions over your data. They send that data to the model by design:
  • Search chat and the answer endpoint;
  • the Assistant reading query results or a Live Read preview;
  • the Assistant or the recovery agent reading a run’s status, logs or job details with their tools (those records keep the full error text);
  • tap script generation, review and diagnosis, and fix suggestions for failed tap runs, which see a tap’s captured output or the source’s documentation rather than pipeline rows.
For these, field protection is the control: protected columns are already ciphertext, tokens or masks by the time they are read. For anything else in those flows, use an AI provider you have an agreement with for that data. For every place each kind of data can go, including embeddings and connected agents, see Data Flows.

​
Not yet supported

  • Format-preserving encryption and tokenization.
  • Nested JSON keys.
  • XML sources.
  • Vector destinations.