Tracking Agent docs
Tracking Agent data and keys: what Datalyze sees
Datalyze sees one thing when the Tracking Agent runs: that your workspace key was used, and when. Your code, your analytics data and your model and analytics keys stay on your runner and with your own providers. Reviewers' GitHub tokens pass once through Datalyze's sign-in step, then stay in their browser and go only to GitHub.
What does Datalyze see when the Tracking Agent runs?
| What | Where it goes | Does Datalyze see it? |
|---|---|---|
| Your code | Your GitHub runner or laptop, and your model provider | No |
| Model calls | Your provider, with your key or your cloud identity | No |
| Analytics data | Read from your tool with your access, into a snapshot on the runner | No |
| Google Sheet updates | Your Sheet, through its receiver or your service account | No |
| Secrets typed into setup | Encrypted in your browser, then saved to GitHub | No |
| A reviewer's GitHub token | The reviewer's browser tab, then api.github.com | Passes once through Datalyze's server at sign-in; never stored |
| Licence check | Datalyze: your workspace key in, the signed method out | That the key was used, and when |
The review, setup and GitHub sign-in pages load no analytics and no third-party scripts, ours included, so nothing typed or shown on them, such as your licence key or the code in a proposal, reaches a tracking tool.
What does the Tracking Agent's licence check send?
The runner sends only your workspace key. It gets back whether the licence is active, your plan, and the method bundle: the written rules and instructions the agents follow. No code, repo name or analytics data is sent. The playbook chapter on analytics in the age of AI explains why an agent needs written rules to be trusted.
The method bundle is versioned and signed. The runner checks its Ed25519 signature before using it and caches a verified copy in .datalyze/cache, so a run still works when Datalyze can't be reached. Set licence.pin in .datalyze/config.yaml to a version such as 2026.10.3, and results never change without you knowing.
Where do the Tracking Agent's keys live?
| Key | Where it lives | What reads it |
|---|---|---|
Tracking Agent key (DATALYZE_KEY) | A secret in your repo. Datalyze keeps only a SHA-256 hash and shows the key once | Your workflow runs |
| Model key | A repo or organisation secret; none at all with a Bedrock or Vertex cloud identity | Your workflow runs |
| Analytics credentials | Secrets in your repo | Your workflow runs |
| GitHub App private key | DATALYZE_APP_PRIVATE_KEY in your repo | Your workflow runs |
| A reviewer's GitHub token | The browser tab's session storage, cleared when the tab closes | That tab, to call api.github.com |
Your own checks and the agent's generated tests run without the runner's secrets: no model keys, tokens, App key, Datalyze key or cloud credentials. GitHub explains how Actions secrets are encrypted and exposed to workflows.
What happens to the analytics data the agent reads?
It goes into a data snapshot, a JSON file on the runner where every figure carries the query that produced it. For properties whose names look like personal data (email, phone, name, query, prompt and similar), only the fill rate and type are recorded, never the values. The snapshot goes to your model provider with the rest of the context, and nowhere else.
Can anyone forge an approval on a pull request?
No. Approvals come only from /datalyze comments, and the runner checks the GitHub account that posted each one against reviewers in .datalyze/config.yaml on the pull request's base branch (or the default branch when the base has none), so a pull request can't change who may approve it. The check keeps its state in a hidden block of its own summary comment, signed with your Tracking Agent key (or DATALYZE_STATE_SECRET), and reads it only from the account the runner posts as. Editing an old comment never adds a command. A /datalyze comment from anyone who isn't a reviewer, a maintainer or the author of a pull request from this repo starts nothing. A pull request from a fork is analysed but never implemented here: implementing runs its install, checks and tests, which would run with your repo's secrets.
Can anyone fake the datalyze/tracking check?
Setup makes both checks required with a repository ruleset on your default branch. With your own GitHub App, the ruleset accepts datalyze/tracking and datalyze/rules only when that App posted them, so no other token can turn them green. Without an App, any token with write access to the repo could post them.
One risk is left either way. Anyone who can push a branch that changes .github/workflows/ can run code with the repo's secrets on that pull request, the App's key included. Protect .github/workflows/ and .datalyze/ with CODEOWNERS and turn on required code-owner review for your default branch, so a change to either can't merge without its owner's review, and give write access only to people you'd trust with those secrets.
Questions about data and keys
Does Datalyze store our code or tracking plan?
No. Both stay in your repo. Datalyze stores your account, your workspace, a hash of your Tracking Agent key, when it was last used, and short-lived rate-limit counters.
Can we keep our model key out of GitHub entirely?
Yes. With AWS Bedrock or Google Vertex the workflow signs in through GitHub's OIDC identity, so no model key is stored anywhere. See models.
What happens if Datalyze is down?
Runs use the verified copy of the method cached in .datalyze/cache. With no cached copy, both checks pass with a note that the pull request wasn't checked, so an outage never blocks a release.
How is this different from hiring Datalyze?
The Tracking Agent is software that runs in your accounts, and Datalyze sees only the licence check. When you hire us for an analytics audit, people on our team work with the access you choose to give them.