Tracking Agent docs

How the Tracking Agent verifies tracking changes

The Tracking Agent verifies every tracking change in four layers before pushing it: scripts, your own checks, generated tests, and an independent verifier. The verifier starts fresh, with no access to the first agent's reasoning, and code that fails verification is never pushed. A fifth layer, a live test that walks the change in a real browser, is optional and turned on per repo.

Generated tests run on Node's built-in test runner in the run's workspace, or for a mobile app in its own test framework. They aren't committed: they're evidence, listed in the report.

It checks coverage, what broke, whether each new event should exist, correctness (example values checked character by character against the code, and the properties sent equal to the properties listed), description quality and personal data. With a data snapshot it also checks that new values match the format of existing data, such as Pro against pro, and that a modified event's history stays comparable.

Each finding is a blocker, a fix or a question. Blockers and fixes go into the fix loop; questions go to the reviewer. Where possible the verifier runs on a different model from the implementer: with Anthropic's defaults, Opus checks Sonnet's work.

1. The failure is sorted. An environment failure (dependencies didn't install, a build or a test login failed) is retried once, and the agent never edits code for it. If it still fails, the status becomes "Can't verify: environment" with what to fix. A check that was already failing before the change, with the same errors, counts as pre-existing.
2. A tracking failure goes to the fix loop, at most two rounds. The implementer gets the evidence and fixes it, then every layer runs again with a new verifier that has no memory of earlier rounds. Each round posts a comment saying what failed and what changed.
3. Still failing, or the fix needs a judgement call, the status becomes "Needs a decision" and the reviewer gets the question with lettered options. For example: "plan doesn't exist yet when signup_started fires. A: fire later. B: drop the property. C: set it as a user property after account creation."
4. The reviewer answers with /datalyze choose p1 B, drops the item with /datalyze drop p1, or accepts it as implemented, in which case the report records who accepted which findings.

If the implementer disagrees with a finding, it must quote the finding and the item goes straight to a person. Nothing is dropped silently.

A generated test can only check a call it can reach. On both of this site's real pull requests, the tracking calls sat inside React components, so the generated tests couldn't trigger them and the verifier checked those items by reading the code instead. A test that captures nothing is reported as not exercised, not as a failure. Moving call sites to the typed tracker makes more of them reachable, and the optional live test reaches them by clicking through the real page.

The playbook chapter on event verification covers the checks a person should still make once data arrives.

1. Install Playwright with Chromium in your repo. The runner uses the repo's own Playwright, or its own if the repo has none.
2. Add live_test to .datalyze/config.yaml with enabled: true.
3. For dev server mode, set start.command, with {port} where the port goes. For preview mode, leave start out; the preview URL comes from GitHub or from preview_url.
4. If flows need to sign in, save a test account in two environment variables and name them under account. Flows type {{username}} and {{password}}, and the model never sees the values.
5. If your app sends analytics through a first-party proxy, list those paths under capture_paths, so the test reads them too.

npm i -D playwright && npx playwright install chromium

# .datalyze/config.yaml
live_test:
  enabled: true
  mode: dev_server          # or preview; default: dev_server when start is set, else preview
  start:                    # dev_server mode
    command: npm run dev -- --port {port}
    ready_path: /
    timeout_seconds: 240
    unset_env: [MIXPANEL_PROJECT_TOKEN]   # removed so a server-side SDK can't send real events
  preview_url: https://app-git-{branch}-team.vercel.app   # optional; {pr} {sha} {branch}
  wait_minutes: 10          # preview: how long to wait for the commit's deployment
  headers: { x-vercel-protection-bypass: $VERCEL_AUTOMATION_BYPASS_SECRET }
  account: { username_env: TEST_USER, password_env: TEST_PASSWORD }
  capture_paths: { /mp/api: mixpanel }
  block_hosts: []
  flush_ms: 6000
  max_flows: 8

headers are sent only to the app's own origin, and a value starting with $ reads an environment variable. block_hosts adds hosts to the ones blocked by default, flush_ms is how long to wait after each flow for batched SDK sends, and max_flows caps the flows planned per run. The reference lists every key with its default.

1. A model plans the flows. The main model reads the code, the routes and the registry and plans a browser flow for the approved items, with steps such as open a path, click a button by its visible name, fill a labelled field, wait for text, scroll, or call the app's own API for a server event. Steps are data, checked by code, and only paths on the app's own origin are allowed.
2. Items it can't reach are listed as "not walked", with the reason: webhooks, email links, third-party redirects such as Calendly or Stripe, and paid plans.
3. Headless Chromium walks each flow through Playwright, in a fresh browser context with clipboard access granted.
4. The browser sends nothing to your analytics project. Every Mixpanel, PostHog and Amplitude request it makes, and every path in capture_paths, is read and answered in the browser, and the typed tracker is switched to capture mode in the page. Ad and tracker hosts are blocked (Google Ads and Analytics, DoubleClick, Meta, LinkedIn, Bing, Clarity, Hotjar, Segment, RudderStack, Heap, FullStory), so a test booking never counts as a conversion. A preview deploy's server is real, though: see the next question.
5. Code compares what was captured with the registry, item by item, using the checks below.

• The event fired.
• Every required property is present, and no property outside the plan was sent.
• Values are inside the registry's examples.
• No value is shaped like an email or phone number.
• A server event has a user id.
• A deprecated event no longer fires, or still fires only inside its dual-fire window.

A flow that stopped before reaching the code, such as a button it expected that wasn't there, makes its items "not exercised", not failed. The report comment and the sign-off screen show expected against captured for each item, labelled "Live test" or "Generated test", and list the flows walked.

In dev server mode, server events are captured when the server uses the typed tracker: the runner sets DATALYZE_CAPTURE and a capture file for the server process. A legacy server SDK isn't captured, so list its variables in unset_env and it can't send. The dev server runs without the runner's secrets, such as model keys and tokens, so it can't create real side effects through them.

In preview mode, the URL comes from the deployments GitHub records for the commit, which Vercel and Netlify post, or from preview_url. Production deployments are never used. A protected preview needs a bypass header in headers; on Vercel that's Protection Bypass for Automation.

It reports "couldn't run (environment)" and never blocks, because the layer is optional. The causes: Playwright isn't installed, Chromium won't start, the dev server didn't come up, or the preview can't be reached, is protected, or never deployed. Troubleshooting has the fix for each.