Tracking Agent docs

The Tracking Agent PR check: every status and what to do

The Tracking Agent posts two required checks on every pull request: datalyze/rules, which runs scripts only, and datalyze/tracking, which runs the agent. Most pull requests need no tracking change and go green with a one-line reason. When one does, datalyze/tracking waits for a person to approve the proposal, then to sign off the verified result.

The statuses are commit statuses tied to one commit. A pending status can wait days for a person without any job running.

datalyze/rules runs three scripts and no model, so it gives the same answer every time and costs nothing:

• datalyze validate --since-baseline: the registry's rules, reporting only findings added since the baseline.
• datalyze scan --fail-on-drift on the files the pull request changed: the code against the registry, such as a live event with no call, or an SDK call outside the adapter.
• datalyze generate --check: the typed tracker matches the registry.

It runs on every push, including the agent's own commits.

Setup makes both checks required with a repository ruleset on your default branch. With your own GitHub App, the ruleset accepts datalyze/tracking and datalyze/rules only when that App posted them, so no other token can turn them green. Without an App, any token with write access to the repo could post them.

One risk is left either way. Anyone who can push a branch that changes .github/workflows/ can run code with the repo's secrets on that pull request, the App's key included. Protect .github/workflows/ and .datalyze/ with CODEOWNERS and turn on required code-owner review for your default branch, so a change to either can't merge without its owner's review, and give write access only to people you'd trust with those secrets.

1. Triage. Scripts decide when the answer is certain; otherwise a small model decides whether the change touches tracking at all.
2. Change map. The agent lists every tracking call the change can reach: calls on changed lines, in touched files, one import level up and down, and on the same funnel.
3. Propose. Four specialists work in parallel (coverage, breakage, dead items, conventions), then a critic merges their items, drops any without evidence, and applies the event budget.
4. Review. A reviewer approves, edits or rejects each item. See reviewing.
5. Implement. The agent writes the approved items into the registry, regenerates the typed tracker and edits the call sites. A scope guard reverts edits outside the approved set.
6. Verify. Scripts, your own checks, generated tests and an independent verifier. See verification.
7. Commit and report. One commit, datalyze: implement tracking changes (round N), and a comment with the results per item.
8. Sign off. A reviewer signs off and datalyze/tracking turns green.
9. After merge. The Google Sheet is rebuilt from the registry on your default branch. The after-merge job moves what shipped from planned to implemented and marks signed-off proposals, in one pull request from the branch datalyze/after-merge. Only statuses and code_refs change in it, so the check passes it without a model call; merge it when convenient.

The PR check catches tracking problems before they ship. The playbook chapter on data accuracy as a system covers the alerts that catch them after.

A new push is triaged on what changed since the last check. If it can't affect tracking, the proposal, approvals and sign-off carry over to the new commit. Once the change is verified or signed off, any push that touches the tracking plan or a file with a tracking call is checked again, without asking a model whether it matters. If it can, the change is analysed again, and decisions on items whose content didn't change are kept. The agent recognises its own commits and doesn't start over for them.

Comment /datalyze skip "reason". Only people in maintainers in .datalyze/config.yaml can skip; it defaults to your reviewers. A skip covers the commit it was given on, is logged in .datalyze/decisions.yaml, and the next audit flags it.

Both checks pass with a note that the pull request wasn't checked. The same happens with an unknown key, a method bundle whose signature doesn't verify, or Datalyze unreachable with no cached method. A lapsed payment never blocks a release.

Not by default. A draft shows datalyze/tracking as pending with a note, and is checked once it's marked ready for review. Set check_drafts: true in .datalyze/config.yaml to check drafts too.

Model spend goes to your own provider, and every comment ends with the models used, the tokens and the cost. When a script can tell a pull request needs no change, no model runs. On this site's repo, a run where triage found no tracking change cost $0.20, and two pull requests that went through the whole flow cost about $1.10 and $3.20.

model.max_cost_usd in .datalyze/config.yaml stops a run once its spend passes the cap. The default is $10. Calls whose price isn't known (Bedrock, Vertex, a gateway, OpenAI or Gemini without model.pricing) count against the cap at $5 per million input tokens and $25 per million output tokens, so the cap still stops a run; the report still says their cost is unknown.