Data Catalog

A Data Catalog is a named bucket that groups related taps and pipelines. It’s pure metadata — assigning a tap to a catalog doesn’t change how the tap runs or where its data lands. Catalogs exist to make a busy platform browsable: instead of a flat list of 40 taps, you see “Stock Prices (12)”, “Weather (6)”, “Earnings (8)” and so on. A Datris catalog is not a Databricks Unity Catalog catalog. On a Databricks pipeline, destination.database.dbName names the Unity Catalog catalog the table lands in; the Datris catalog is a separate grouping label, carried to Databricks as the datris_catalog tag when the Unity Catalog metadata push is on. See Unity Catalog.

​

When to use a catalog

Use one whenever you have more than a handful of related taps that should travel together — same source, same domain, same project. You don’t have to use catalogs. Anything you create outside a catalog falls into the Uncataloged group, which works fine for one-off taps and pipelines.

​

How catalogs are stored

Catalogs aren’t a separate table or external system. Each tap and pipeline carries a nullable catalog string field; the platform groups by that field. Two consequences:

• A catalog “exists” as soon as anything is assigned to it. There’s no separate create step required to use a name.
• Renaming a catalog means updating the catalog field on every tap and pipeline in it. The Catalog tab’s rename control does this in one step, or call rename_catalog over MCP.

When you create a catalog explicitly via the UI, Datris persists the name by writing a placeholder tap named __catalog__<name> (description “Catalog placeholder”, enabled=false). This makes empty catalogs survive across UI reloads. The placeholder is hidden from the Taps list and ignored by the runtime.

​

The Catalog tab

The Catalog tab in the UI shows one card per catalog plus an Uncataloged card.

• Each card shows the catalog name and counts of taps + pipelines.
• Rename catalog — the pencil icon on a named catalog’s card turns the name into an editable field. Press Enter (or the check mark) to save, Escape to cancel. The new name may use letters (upper or lower case), digits, _ and - only; anything else is converted when you leave the field (spaces become _, other characters are dropped). Capitalisation is kept. Every tap and pipeline in the catalog moves to the new name in one step.
  • A rename cannot target a catalog that already exists: it is refused and both catalogs stay as they were. To combine two catalogs, use Move all contents (the move icon on the source catalog’s card) instead.
  • If some items could not be moved, the page lists each one with the reason; the rest are renamed. If the catalog was renamed or deleted from somewhere else after the page loaded, the rename or delete changes nothing, says the catalog was not found, and refreshes the list. An item that someone moved to another catalog while the rename ran is reported as no longer in the catalog; refresh the page to see where it is now.
  • API keys whose capabilities are scoped to the old catalog name keep that scope and no longer match the renamed items. After such a rename a banner names those keys; update them under Configuration → Keys.
  • Catalog names may be mixed case (for example DatrisFund or Sales_Q3), and they are matched case-sensitively. A rename to a name that differs from another existing catalog only by case (for example datrisfund when DatrisFund exists) is refused with a message that the catalog already exists with different capitalisation, because it would create a second catalog next to it. A catalog can still be renamed to a different capitalisation of its own name.
• Click a card to expand it. Inside you’ll see:
  • Each tap with its enabled/disabled status, last-run outcome, and record count.
  • Each pipeline with a quick-jump link to its config.
• + Create Catalog — make a new empty catalog. The name is sanitized to letters, digits, _ and - (spaces become _); capitalisation is kept. A name that differs from an existing catalog only by capitalisation is refused, here and in the new-catalog field of the tap and pipeline forms; typing an existing name exactly in those forms selects that catalog.
• Delete catalog, keep items — the delete icon on a named catalog’s card opens a confirm with two choices. Keep items (move to Uncataloged) is the default: the catalog is removed and every tap and pipeline in it moves to Uncataloged. Nothing is deleted.
• Delete catalog, delete items and their data — the second choice deletes the catalog plus every tap and pipeline inside it, including the data already written to their destinations. This is destructive and not reversible, so Delete stays disabled until you type the catalog name exactly (case-sensitive). If some items could not be deleted, the page lists each one with the reason. With GitHub script storage, a script file that was edited in the repository since Datris last wrote it is kept (the item itself is still deleted), and the page names the file in a warning banner.
• Download catalog — the download icon on a named catalog’s card saves the catalog as a zip archive (see What an archive contains). The archive holds no secret values and no data, so it is safe to keep alongside other configuration backups.
• Import catalog — the Import catalog button next to + Create Catalog restores an archive. Pick the zip (the panel shows its name and size), enter the catalog name to restore into (it defaults to the file name), and choose:
  • Create new catalog (refuse if it exists) — the default. The restore is refused when the catalog already exists, when any archived tap or pipeline name already exists on this install, or (with GitHub script storage) when an AI script file of a new pipeline differs at branch head; the page lists each clash and nothing changes. A restore never merges into an existing catalog.
  • Overwrite catalog — replaces the items already in that catalog. Restore stays disabled until you type the catalog name exactly.
  • Skip incremental tap state (for a clone) — restores the taps without the source’s incremental cursor, so a copy starts from scratch.
  After the restore the page shows how many taps and pipelines came back, names any secrets they need that do not exist yet, lists items that could not be restored with the reason, and expands the restored catalog. Restored taps are disabled: test each one, then enable it. When some restored taps read tables produced by other restored taps, the result says how many: run their sources first, in the enable order.
• Enable order hints — in an expanded catalog, each disabled tap shows its level in the catalog’s enable order (Level 1 reads no other tap’s tables), or needs: x, y while the taps it reads from have not landed data. The Disabled badge menu names the sources to run first and offers Run now for a tap whose last test passed: one real run that lands its data without enabling it.
• Enable in order — on a catalog with at least one disabled tap, the play-list icon on the card shows the catalog’s enable order level by level, and Start enables the whole catalog in that order in one job. For each tap the job runs a test if its last test did not pass, enables it, does one real run and waits until that data has landed before moving to the taps that read it. A tap whose source failed or landed nothing is skipped with the reason, so you see which source to fix. A run that returns no records fails the tap only when another tap in the enable order reads from it; a tap nothing depends on (for example one that is fed through run parameters) stays enabled and is marked landed with “no records returned; nothing depends on this tap”. Progress shows on the card, one row per tap, and Cancel stops the job after the tap in progress (its run is not interrupted; the remaining taps are marked cancelled). While the job runs, the catalog cannot be renamed, deleted or restored over. After a restore, the result banner offers the same Enable in order link when restored taps depend on each other.
• Uncataloged is not a catalog — its card has no rename, delete or download control. Delete uncataloged items one at a time (below).
• Delete individual tap or pipeline — expand any catalog (including Uncataloged) and use the trash icon on a specific row to delete just that item. Clicking the trash icon swaps the row’s action icons for an inline confirm: Delete / × for a tap; Config & Data / Data Only / × for a pipeline (Data Only wipes the rows but keeps the pipeline config so the next ingest fills it again).
• Row actions menu (Uncataloged) — each item in the Uncataloged group has a ⋮ kebab menu after the trash icon with three actions:
  • Edit — opens the tap or pipeline editor.
  • Delete — same as the trash icon (inline confirm).
  • Move to catalog — submenu listing every other catalog. Picking one sets the item’s catalog field and reloads the view. Before persisting, the UI checks the target catalog for a tap or pipeline of the same name; on a clash the move is blocked and a red banner explains the collision (“A tap named ‘foo’ already exists in catalog ‘reporting’. Rename one of them first.”).

​

Assigning a catalog

Four ways to put a tap or pipeline into a catalog:

1. Tap editor — when creating or editing a tap manually, set the Catalog field.
2. Pipeline editor — same field on the pipeline form.
3. CLI — pass --catalog <name> to datris ingest when creating a new pipeline. Re-ingesting an existing pipeline preserves its current catalog (and the rest of the config) — use the set_catalog MCP tool to change a catalog after the fact.
4. MCP agent — call set_catalog(pipeline="<name>", catalog="<label>") or set_catalog(tap="<name>", catalog="<label>") to set the label on an existing item. Pass an empty string (or omit catalog) to clear it back to Uncataloged. New pipelines created via create_pipeline accept an optional catalog argument directly.

Leave the field blank to keep the item Uncataloged.

​

API and MCP

Catalogs are a field on the Tap and Pipeline configs. You read and write single items through the existing endpoints:

• GET /api/v1/taps and GET /api/v1/pipelines return the catalog field on each item.
• POST / PUT to those endpoints accept a catalog field. To move an item, send a request with the new value.

Two endpoints act on a whole catalog:

• PUT /api/v1/catalog/{name} with body {"newName": "<label>"} renames the catalog. If newName already exists the request is refused with 409 and nothing changes; move items to combine catalogs. The response lists the items renamed, any failed items with their error, and affectedKeys: API keys still scoped to the old name.
• DELETE /api/v1/catalog/{name}?mode=detach removes the catalog and moves its items to Uncataloged. mode=cascade&confirm=<name> deletes the items and their data as well; confirm must equal the catalog name. The cascade response’s warnings lists follow-up actions for deleted items: each pipeline whose Unity Catalog table Datris did not drop, as {pipeline, message} with the table an admin has to drop in Unity Catalog, and each script file kept in the code repository because it was edited there since Datris last wrote it, as {pipeline, message} or {tap, message} naming the file. The Data Catalog page shows these in a banner until dismissed. Uncataloged cannot be renamed or deleted.

From an MCP agent:

• set_catalog wraps the read-modify-write on a single pipeline or tap, so you don’t have to round-trip the full config yourself.
• rename_catalog renames a whole catalog in one call, with the same rules and affected-keys behaviour as the Catalog tab: the new name must not already exist.
• delete_catalog removes a catalog and moves its items to Uncataloged. It never deletes items or data; deleting a catalog together with its items is available from the UI or the API only.
• catalog_enable_order returns the catalog’s taps in dependency levels with, per tap, whether its last test passed, whether a real run has landed data and which sources still block it (GET /api/v1/catalog/{name}/enable-order). Read-only. See Dependencies between taps.
• enable_catalog_ordered starts the same job as Enable in order (POST /api/v1/catalog/{name}/enable-ordered, no body; 202 with the job id and plan). It needs tap:run, and the key must be allowed to update and run every tap in the catalog. Progress is the job block of catalog_enable_order (or GET /api/v1/catalog/{name}/enable-ordered); a finished job stays readable for an hour. Cancel with DELETE /api/v1/catalog/{name}/enable-ordered or from the card; there is no MCP cancel tool. Under an agent policy the action is tap:run:enable-ordered, so one approval covers the whole job. How long the job waits for one tap’s data to land is datris.catalog.enableWaitMs (default 30 minutes).
• export_catalog returns a catalog as a zip archive (base64) with its manifest; restore_catalog restores one under the same or a new name, with the same rules as Import catalog. From a terminal, datris catalog export and datris catalog restore do the same (see CLI).

For browsing, list taps or pipelines and group client-side by catalog.

​

What an archive contains

A catalog archive holds:

• the definition of every tap and pipeline in the catalog,
• each tap’s script,
• each tap’s incremental state (its cursor),
• the validation schemas the pipelines reference,
• each pipeline’s AI data-quality and AI transformation scripts, with the model and time they were generated.

A restore makes the archived AI scripts the live scripts, so the target runs exactly what the source ran and no model call is made. A script is regenerated on the target instead when the archive was exported before script records were included (re-export it to carry them), when it was written for a different instruction or source schema, or when you ask for it (datris catalog restore --regenerate-scripts, regenerateScripts=true on the API). When the archived script cannot be used but the pipeline being overwritten already has a script for the same instruction and schema, that live script is kept as it is. With GitHub script storage, a restore that brings a new pipeline whose AI script file already exists at branch head with different content is refused and nothing is written. The restore result lists which scripts were imported, regenerated or kept, with the reason. It never holds secret values, destination data, run history or version history. The archive lists the names of the secrets the catalog needs; create any that are missing on the target install. Restored taps are disabled until you enable them. To promote a catalog from one install to another, for example from dev to QA to production, see Dev, QA and production. Things to know when restoring:

• A restore never merges. Tap and pipeline names are unique across the install, so restoring onto the same install under a new name is refused while the source catalog still exists. Delete or rename the source catalog first.
• Inline REST bearer tokens and API keys on a pipeline are removed from the archive. Re-enter them after the restore; the result names each field that needs it.
• An API key without unscoped secret read access gets a “not checked” warning for each required secret instead of a list of missing secrets.
• To make a clone that starts from scratch, skip the incremental tap state (the checkbox in the import panel, or --no-state on the CLI).

​

What catalogs don’t do

• They don’t isolate access — anyone with API access to a tap can see it regardless of catalog.
• They don’t affect tap or pipeline behavior, scheduling, or destination.
• They aren’t permissioned — catalogs are a flat namespace per tenant.

If you need true tenant isolation, that’s handled at the Datris environment level, not via catalogs. See tenant isolation in the configuration reference.