case-studies

Answering Agent: Helped create a call score using AI

About Answering Agent

Answering Agent is an AI-powered calling platform that automates customer outreach, follow-ups, and support conversations. As an AI-first product, the quality of each call directly impacts customer satisfaction, product trust, and conversion outcomes. But without a clear way to evaluate call performance, the team struggled to understand how well the AI was functioning — and what needed improvement.

Challenge 1: No Visibility Into AI Call Quality

The core issue Answering Agent faced was the inability to evaluate whether an AI-generated call was “good” or “bad,” and more importantly, why.

This created multiple problems:

The team couldn’t pinpoint weaknesses in the AI’s calling logic
Product decisions were based on assumptions rather than data
There was no standardized framework for evaluating calls
Improving the AI became guesswork instead of a structured process

Without call-level insights, the product's evolution was limited.

Solution

To give the team clarity and actionable insights, we built an end-to-end evaluation framework for AI call quality.

1. Deep dive into the product & industry benchmarks

We studied:

How the AI conducted calls
Typical call flows and expected outcomes
Industry standards for conversational AI performance
Real-world call scenarios and edge cases

This groundwork enabled us to define what “good” looked like.

2. Defined a complete set of evaluation metrics

We created a structured metric system that captured dimensions such as:

Call clarity and coherence
Response relevance
Latency and hesitation patterns
Completion of the intended task
User sentiment cues
Compliance with call scripts or guidelines

These metrics formed the foundation of a consistent scoring framework.

3. Built an AI call scoring model

On top of the defined metrics, we developed a model that:

Analyzed each AI call
Scored it across the defined dimensions
Highlighted specific issues when a call underperformed
Provided an overall call quality score

This transformed raw call data into actionable insights.

Result

The scoring system is now actively used across the team:

They can instantly see which calls performed poorly and why
Product and engineering teams can prioritize improvements based on real data
The model enables continuous optimization of the AI calling logic
Over time, call quality has become significantly more predictable and measurable

The team now operates with full visibility into AI performance, making improvements faster, more targeted, and far more effective.

Connecting marketing and product data

The short version Your marketing site and your product live on two subdomains, and most companies never join the data between them. So you can see cost per visitor by channel, but not which channel brings users who signup, activate, retain, and pay. That second number is the one that

D2C Playbook 2 -- Putting all your data in one place

The short version On your own website and app, you know who your customer is. You know that Priya came back three times before she bought, that she found you through an Instagram ad, that she abandoned a cart in March and came back in May. That is the layer

Analytics Debt: The Silent Killer Nobody Talks About Until Series B

Every startup I've worked with has analytics debt. Every single one. Not because they're bad at data. Because they were busy doing the thing that matters more early on. Building a product people want. But here's what happens. You raise your Series A. You

D2C Playbook 1 -- The Data Foundation: Fix Your Owned Layer Before You Scale Spend

The short version Here's the thing nobody tells you when you're about to scale spend. Every channel you're about to pour money into gives you data you can't fully trust. Quick commerce hands you almost nothing about who your customer is. Meta