Tracking Agent docs

How the Tracking Agent audit works

The Tracking Agent audit compares your tracking plan, what your code fires, and what your analytics tool actually received. Every event and user property lands in exactly one bucket, such as healthy, broken, dead, or arriving but not in the plan, and the report gives a health score. Run it with datalyze audit and a data snapshot from your analytics tool.

1. Build a data snapshot with datalyze snapshot mixpanel, posthog or amplitude. The Mixpanel recipe reads Mixpanel's query APIs and a sample from the raw event export. For each tool's credentials, and for other tools, see connecting analytics.
2. Run datalyze audit --snapshot snapshot.json, with a scope option if you want one, and --md report.md to save the report.
3. Read the report, starting with broken and dead items.
4. Optionally run it again with --apply to write the statuses the data proves, plus _audit.json for the Google Sheet's Audit tab.

datalyze snapshot mixpanel --project-id 1234567 --region us --out snapshot.json
datalyze audit --snapshot snapshot.json --md report.md

The install audit runs on its own, on the setup pull request, with the recipe for analytics.tool. After a merge, the post-release watch runs a PR-scoped audit on its own (below). Running a full audit on a schedule or from the review page isn't available yet.

• Properties: a planned property that never arrives, one filled on less than 95% of events (audit.min_fill_rate), one arriving as two types, or values outside the plan.
• Volume: an event that stopped arriving, or whose daily volume over the last 7 days fell below half of what it was before.
• Funnels: more outcomes than intents, such as more order_placed than checkout_started.
• Double-firing: more than 5% of an event's rows repeating for the same user within a second (audit.max_duplicate_rate).
• Personal data: property values shaped like an email or phone number.
• User properties: fill rate, stale values, and set used where set_once was meant.
• Identity: an identified share below 20% (audit.min_identified_share) and duplicate profiles, where the tool reports them.
• Super properties that don't arrive: an event whose properties are all known but lack a super property declared for its side, which usually means a path that skips the wrapper adding it, such as a new typed tracker call next to a legacy wrapper.

The health score is the share of live plan items (implemented or needs_review) that are healthy, counting only items the snapshot measured. Below it, findings are grouped by bucket, each with its code locations and the query behind every figure, so you can check any number yourself.

The audit never adds or deletes registry entries. --apply only changes statuses the data proves: planned and arriving becomes implemented, and implemented with nothing arriving becomes needs_review.

When we ran the install audit on this site, the registry came out at 46 events in 6 area files, with no drift between the plan and the code, and the baseline accepted 41 legacy findings.

datalyze watch checks that the tracking a merged pull request changed arrives as planned. The workflow runs it every day at 06:41 UTC; to run it by hand, open Actions, then Datalyze, then Run workflow, and pick watch.

1. It finds recent changes: pull requests merged into your default branch in the last 14 days that have a proposal file, and in each, the items a reviewer approved (events and user properties). A property item is watched through its event.
2. It waits for traffic: until at least 50 of each watched event have arrived, counted from the merge day, or 48 hours after the merge, whichever comes first. Deprecations and user properties wait the full 48 hours.
3. It runs a PR-scoped audit against a fresh snapshot from the recipe for analytics.tool, or the file given with --snapshot.
4. It reports once: one comment on the merged pull request, final once posted, with each item's result, such as arrives as planned, a deprecated event still arriving, values outside the plan, or a property not arriving. The same results become rows on the Sheet's Audit tab with Source "Watch PR #n", where a person marks each one verified.
5. It opens fix pull requests for the problems, one per area, through the PR check, so both human gates apply.

# .datalyze/config.yaml
watch:
  min_events: 50
  hours: 48
  lookback_days: 14
  open_fix_prs: true   # false: report only

Without data, because no recipe or credentials are set, the watch waits, then says on the PR that it couldn't run yet and why. It tries again on each run while the PR is inside watch.lookback_days, so it reports once the credentials are set; on its last run inside that window it posts that it stopped. A snapshot that fails on a run, for example on a rate limit, counts as no data for that run. For a mobile app the watch waits for the app version that ships the change and counts only that version and newer ones: see mobile apps.

Through the PR check, like any other change, so both human gates apply: a person approves the proposal and signs off the result. You choose which findings to fix, and the agent opens one fix pull request per area.

• In the Google Sheet: put yes (or ✓ or x) in the Fix column of Audit tab rows. The hourly requests job opens one fix pull request per area for the ticked rows and writes its link into Fix PR. Healthy rows and rows that already have a Fix PR are skipped.
• From a laptop: run datalyze fix --items <name,name>. It reads buckets and checks from --audit (default analytics/registry/_audit.json), and --source records where the findings came from, such as "Watch PR #12".
• From the watch: problems it finds become fix pull requests on their own, unless watch.open_fix_prs is false.

A fix pull request is a proposal pull request, like one from the Requests tab: the request agent proposes the items, code checks them, the proposal is signed, and the PR check adopts it only with that signature. Then a reviewer approves and signs off as usual. The playbook chapter on data accuracy as a system covers keeping numbers right between audits.