Tracking Agent docs

Tracking Agent troubleshooting and FAQs

Q: How do I re-run the Tracking Agent check?

Comment /datalyze retry on the pull request. Reviewers and the pull request's author can use it after an error, an environment failure, or a run stuck on analysing or implementing.

This page lists Tracking Agent problems by symptom, each with its cause and the file, secret or command that fixes it. Start with the table, then read the section for your symptom. The agent also says what went wrong on the pull request itself, in its status line and its comment.

By Ansh AgrawalUpdated 20 Oct 2026

What's wrong, and how do I fix it?

Symptom	Most likely cause	Fix
`/datalyze` comments do nothing	The workflow isn't on the default branch yet	Merge the setup pull request
The agent replies that you aren't a reviewer	Your login or team isn't in `reviewers`	Add it to `.datalyze/config.yaml` in a pull request; it counts once merged
Your CI never runs on the agent's commit	No GitHub App, so the default token pushed it	Create the App in setup or with `datalyze github-app create`
Both checks pass with "not checked"	The licence is inactive, or `DATALYZE_KEY` is wrong	Renew, or save the current key
"Can't verify: environment"	Dependencies, a build or a test login failed	Set `install`, fix the environment, `/datalyze retry`
The check errors with "No model key"	The model key secret is missing or misnamed	Add the secret the error names
`datalyze/rules` fails	The code and the registry disagree	Read its findings; see the table below
A plain `/datalyze approve` leaves items pending	Breaking, deprecation, money or lower-confidence items	Approve them by name: `/datalyze approve p2,p4`
An approval is refused because the proposal changed	The proposal file changed after you opened the review page	Reload the review page, read the changes, approve again
The live test says "couldn't run (environment)"	Playwright or Chromium is missing, the dev server didn't come up, or the preview can't be reached, is protected or never deployed	See the live test section below; it never blocks the check
The watch hasn't reported on a merged pull request	It's still waiting for 50 of each watched event or 48 hours, or it has no data	Wait, or check the snapshot recipe's credentials

Why do /datalyze comments do nothing?

GitHub runs workflows for issue_comment events only from the workflow file on your default branch. Until the setup pull request merges, comment commands on other pull requests start nothing. Other causes:

The command was added by editing an old comment. Post a new comment.
The command isn't at the start of its own line.
The comment came from a bot account. Bot comments never start a run.
GitHub Actions is turned off for the repo or the organisation.

Why is datalyze/tracking stuck on pending?

Usually it's waiting for a person, which is pending by design: read the status line, which says what it's waiting for (review, a decision or sign-off). A draft pull request also stays pending until it's ready for review, unless check_drafts is on. If the status still says it's analysing or implementing long after the run ended, open the workflow run in the Actions tab to see why, then comment /datalyze retry: from analysing it starts the analysis again, and from implementing it runs implementation and verification again.

Why doesn't our CI run on the agent's commits?

Commits pushed with the default Actions token don't start other workflows, as GitHub documents. The agent's report says when that happened. Create your own GitHub App in setup, or run datalyze github-app create, so DATALYZE_APP_ID and DATALYZE_APP_PRIVATE_KEY are saved and later commits are pushed with the App's token.

Why did the check pass with "not checked"?

The runner couldn't confirm an active licence, so it let the pull request through rather than block it. The causes: the subscription has lapsed; DATALYZE_KEY is missing, mistyped or a key you've since rotated; the method bundle's signature didn't verify; or Datalyze couldn't be reached and there was no cached method. Check the plan on your Tracking Agent dashboard, and if you rotated the key there, save the new one as DATALYZE_KEY.

What does "Can't verify: environment" mean?

Something other than the tracking failed: installing dependencies, a build, or a test login. The agent retried that layer once and won't edit code for it. The most common cause on GitHub Actions is a missing install: on a comment event the job's checkout is the default branch, so set install: npm ci (or your lockfile's equivalent) in .datalyze/config.yaml. Fix the cause, then comment /datalyze retry.

Why does datalyze/rules fail?

Finding	What it means	Fix
A live item with no call	The code that sent it was removed, but the registry still lists it as live	Deprecate it in the registry, with a reason
A call that isn't in the registry	New tracking added without a plan entry	Add the entry, or let the PR check propose it
SDK call outside the adapter	Code calls the analytics SDK directly	Call the typed tracker, or add a reviewed `direct_sdk_exceptions` entry
Event name the scanner can't read	The name is built at runtime	Use a literal name or a typed tracker method
Email or phone number as the user id	identify receives personal data	Identify with an internal id
Typed tracker out of date	The registry changed without regenerating	Run `datalyze generate` and commit the result
A rule finding from `validate`	A name, type, example or required field breaks a rule	Fix the entry; the finding says which rule

Problems that were there at install are in _baseline.json and don't fail the check; only new ones do.

Why does the check say "No model key"?

The runner looked for the model key in the variable for your provider, such as ANTHROPIC_API_KEY, and found nothing. Add it as a repository or organisation secret with that name, or, if your organisation secret has another name, map it in the workflow's env block. With Bedrock or Vertex there's no key: check the cloud sign-in step in the workflow instead.

Why does the live test say "couldn't run (environment)"?

The problem is the test environment, not your tracking, and the live test is optional, so it never blocks the check. Fix the cause for next time:

Playwright or Chromium is missing: run npm i -D playwright && npx playwright install chromium, or install them in the workflow.
The dev server didn't come up: check live_test.start.command (with {port}) and ready_path, and raise timeout_seconds (default 240) if the app is slow to start.
The preview is protected: add a bypass header in live_test.headers. On Vercel, that's the Protection Bypass for Automation secret.
The preview can't be found or never deployed: the runner uses the deployment GitHub records for the commit, which Vercel and Netlify post, and never a production deployment. Set preview_url if your host doesn't post one, or raise wait_minutes (default 10) if deploys take longer.

Why hasn't the watch reported on my merged pull request?

The watch reports once enough traffic has arrived: at least 50 of each watched event since the merge day, or 48 hours after the merge, whichever is first. Deprecations and user properties always wait the full 48 hours. It only looks at pull requests merged into your default branch in the last 14 days that have a proposal file, and only at items a reviewer approved.

If it has no data, because no recipe or credentials are set for analytics.tool, it waits, then posts once on the pull request that it couldn't run and why. It runs daily at 06:41 UTC; to run it now, open Actions, then Datalyze, then Run workflow, and pick watch.

Why do a mobile app's generated tests capture nothing?

The capture file reaches the test only if the test command passes the environment on. Gradle does, but it skips a test task it thinks is up to date, so keep --rerun in tests.native.run. xcodebuild passes on only variables starting with TEST_RUNNER_, which the runner sets for you. A test that ran but captured nothing is reported as not exercised, which usually means the code it calls goes to a singleton SDK object the test can't replace.