Repository storage is opt-in and coexists with built-in storage. Existing taps stay where
they are until you move them; you can run a mix of both indefinitely, and move any tap back to
built-in storage at any time.
What you get
- A commit per save — from the tap wizard, the Assistant, or the API. Commit messages are
templated (e.g.
tap(weather-forecast): update via Datris). - Runs pinned to a commit — a scheduled run executes exactly the version that was saved, served from a local cache, so cron runs make no GitHub API calls and keep working if GitHub is unreachable.
- External editing — commit changes from your IDE; Datris detects the drift when the tap is opened and offers to load the latest version.
- Conflict protection — if the script changed in the repository after you opened it, saving is rejected with a clear message instead of silently overwriting.
Setup
1. Create (or pick) a repository
Any repository works — private is typical. Scripts are written to<path prefix><tap-name>.py (default taps/<tap-name>.py) on the branch you configure.
2. Create a fine-grained personal access token
A PAT is created under a GitHub user account and scoped to repositories — you don’t add it to the repository itself. Datris needs Contents: Read and write on just the one repo:- On GitHub, click your avatar → Settings → Developer settings (bottom of the left sidebar) → Personal access tokens → Fine-grained tokens → Generate new token.
- Set a name and an expiration.
- Resource owner: pick the user or organization that owns the target repository.
- Repository access: choose Only select repositories and pick the repo.
- Under Permissions → Repository permissions, set Contents: Read and write (Metadata: Read is added automatically as a mandatory permission).
- Generate the token and copy it immediately — GitHub shows it only once.
Fine-grained PATs always expire. When the token lapses, saves and non-cached reads for
repository-backed taps fail until you update the secret with a fresh token. Put the expiry
date somewhere you’ll see it.
3. Connect it in Datris
In Configuration → Code Repository:- Enter the repository (
owner/repo), branch, and path prefix. - Under Token secret, click Create one, name it (e.g.
github), and paste the token. This stores it in the secrets store with the shape Datris expects — atokenfield, tagged_type=repo_token. - Click Test connection — it verifies the token can reach the repository, has push access, and that the branch exists.
- Check Enable repository storage and Save. Saving re-validates the connection.
GitHub Enterprise Server
Set API base URL to your instance’s API endpoint (e.g.https://ghes.example.com/api/v3).
Leave it at https://api.github.com for github.com.
Day-to-day behavior
Run history records the commit SHA each run executed, so any run can be traced to the exact
script version in your repository.
Moving taps between backends
In the Taps list, each row shows a GitHub badge when its script is repository-backed. The cloud action on the row moves a tap’s script in either direction:- Move to GitHub — commits the current script to the repository and re-points the tap.
- Move to built-in storage — copies the script back to object storage. The repository file is left in place (remove it manually if you want it gone).
Pipeline scripts
The Python scripts behind a pipeline’s AI data-quality rule and AI transformation use the same repository. With repository storage enabled, the script generated for a new pipeline (or for a pipeline that has no script yet) is committed instead of going to built-in storage. Path layout. Each pipeline gets a folder under the configured path prefix, on the configured branch. With the default prefixtaps/:
taps/<tap>.py) are untouched, and the token needs no new permission.
Commit messages. {name} is <pipeline>/<kind> (orders/dataQuality,
orders/transformation), {action} is create, update or delete, and {user} is the label
of the API key that saved the pipeline, or datris when there is none (the script was generated during a run, or saved from a UI session without an API key).
With the default template (or an empty one) pipeline scripts use
pipeline({name}): {action} via Datris, for example pipeline(orders/transformation): update via Datris.
A custom template is applied as written, so a template written for taps (for example one starting
tap() also prefixes pipeline commits that way.
Pinned runs. The pipeline records the commit SHA of each script. Runs read that commit from
the local cache, so a run makes no repository call once the commit is cached, and the run’s
status line names the short SHA (“Running stored CodeGen script generated at … (code repository
commit abc1234)”). GET /api/v1/pipelines/{name}/codegen-scripts returns storage: "github",
repoPath and commitSha for each script.
Editing in the repository. Runs stay on the recorded commit after an edit in GitHub. The
read endpoint then reports drift: true with the branch-head headSha, and for the transformation script the Columns section
of the lineage panel shows “Changed in the repository” with a
Load latest action. To adopt the edit, call
POST /api/v1/pipelines/{name}/codegen-scripts/{kind}/pull (needs pipeline:update); it records
the head commit and makes no commit of its own. Later saves keep the adopted edit while the
instruction and source schema are unchanged.
Conflicts. When the instruction or schema changes, the save regenerates the script and commits
it on top of the recorded commit. If the file was edited in the repository since then, the commit
is rejected: the pipeline is saved, the script and its recorded commit are unchanged, and the save
response carries a warning naming the two ways forward: pull the edit, or replace it with
POST /api/v1/pipelines/{name}/codegen-scripts/{kind}/regenerate?overwrite=true. A forced regenerate
without overwrite=true is rejected the same way (409). A hand edit is never overwritten silently;
an overwritten edit stays in the repository history.
Until the conflict is resolved, runs keep executing the recorded commit, without the new instruction
or schema and with no model call. The run’s status line names the commit and says to pull or
regenerate with overwrite=true, and the read endpoint shows conflict: true with the
conflictReason. Saving the same configuration again makes no model call and repeats the warning.
Pull or a regenerate with overwrite=true clears the conflict. A run that meets the same conflict
(for example after a version restore) records it the same way and runs the recorded commit.
Moving a script. An existing script keeps its storage on every save. To move one, call
regenerate with storage=github or storage=builtin; the old copy is left in place. Scripts that
existed before repository storage was enabled stay in built-in storage until you move them.
Leftover copies are not tracked. Deleting the pipeline removes only the copies its scripts point at
now, so files and objects left behind by earlier moves stay until you remove them. Moving a script
back to the repository writes over a leftover file at the same path without a conflict check, and
the commit is labelled create.
When the repository is unreachable or the token has expired:
- At save, the pipeline is saved and the script is pending with the reason; the save response carries a warning. The first run after the repository is back generates and commits it.
- A run whose recorded commit is cached is unaffected.
- A run whose commit is not cached (the cache does not survive an image rebuild) fails with “Code repository is unreachable and no cached copy … exists locally”. It does not fall back to generating a different script.
- A run that generates a pending script and cannot commit it runs the generated script, logs a warning, and the script stays pending.
storage=builtin to move the script back to built-in storage.
Delete. Deleting the pipeline (or a catalog together with its items) makes a delete commit per
script file that is unchanged since its recorded commit; git history is kept. A file that was edited
in the repository since then is kept: a hand edit is never removed by Datris. The pipeline and its
script records are still deleted, and the delete response’s warnings names each kept file so you
can remove it by hand if it is no longer wanted. When the repository cannot be reached the files are
kept the same way, with a warning, and the delete still goes ahead. If the delete request itself fails
(for example it times out), the warning says the delete could not be confirmed: check the repository
and remove the file by hand if it is still there. Deleting a tap whose script is in
the repository follows the same rule. Saving a pipeline without an AI instruction it had before
removes that script the same way: an edited file is kept and named in the save response’s warnings.