Skip to main content

Support agent analytics: Track quality and ROI

You can't improve what you don't measure. I've worked with teams that shipped support agents without any metrics, ran them for months, and had no idea if they were helping or hurting. The result: half a million dollars spent, zero improvement. This article covers production-grade analytics for support agents: which metrics matter, how to collect them, what targets to aim for, and how to build improvement loops that actually work.

Core support metrics: What to measure​

Not all metrics matter equally. Focus on these five:

from dataclasses import dataclass
from datetime import datetime
from typing import Optional

@dataclass
class SupportMetric:
    """A single metric data point."""
    metric_name: str
    value: float
    timestamp: datetime
    conversation_id: str
    customer_id: str
    dimension: dict = None  # intent, tier, language, etc.

class MetricsCollector:
    """Collect and aggregate support metrics."""
    
    def __init__(self):
        self.metrics = []
    
    def record_resolution_time(
        self,
        conversation_id: str,
        customer_id: str,
        time_minutes: float,
        resolved: bool
    ):
        """Record how long it took to resolve (or attempt) an issue."""
        self.metrics.append(SupportMetric(
            metric_name="resolution_time",
            value=time_minutes,
            timestamp=datetime.now(),
            conversation_id=conversation_id,
            customer_id=customer_id,
            dimension={"resolved": resolved}
        ))
    
    def record_conversation_turns(
        self,
        conversation_id: str,
        customer_id: str,
        num_turns: int
    ):
        """Record how many turns a conversation took."""
        self.metrics.append(SupportMetric(
            metric_name="conversation_turns",
            value=num_turns,
            timestamp=datetime.now(),
            conversation_id=conversation_id,
            customer_id=customer_id
        ))
    
    def record_csat(
        self,
        conversation_id: str,
        customer_id: str,
        score: int  # 1–5
    ):
        """Record customer satisfaction score (post-interaction survey)."""
        if not 1 <= score <= 5:
            raise ValueError("CSAT must be 1–5")
        
        self.metrics.append(SupportMetric(
            metric_name="csat",
            value=float(score),
            timestamp=datetime.now(),
            conversation_id=conversation_id,
            customer_id=customer_id
        ))
    
    def record_escalation(
        self,
        conversation_id: str,
        customer_id: str,
        escalated: bool,
        reason: Optional[str] = None
    ):
        """Record whether conversation escalated."""
        self.metrics.append(SupportMetric(
            metric_name="escalation",
            value=float(escalated),
            timestamp=datetime.now(),
            conversation_id=conversation_id,
            customer_id=customer_id,
            dimension={"reason": reason}
        ))
    
    def record_cost(
        self,
        conversation_id: str,
        customer_id: str,
        cost_cents: int,  # model inference + tools
        detail: dict = None
    ):
        """Record cost of handling conversation."""
        self.metrics.append(SupportMetric(
            metric_name="cost",
            value=cost_cents / 100.0,  # Store as dollars
            timestamp=datetime.now(),
            conversation_id=conversation_id,
            customer_id=customer_id,
            dimension=detail or {}
        ))

# Usage
collector = MetricsCollector()
collector.record_resolution_time("conv_123", "cust_456", 3.2, resolved=True)
collector.record_csat("conv_123", "cust_456", 4)  # 4 out of 5
collector.record_cost("conv_123", "cust_456", 42)  # $0.42

Target metrics and benchmarks​

Here's what production support systems achieve in 2026:

MetricTier-1 AgentsAI Agents (Target)Best-in-Class
Resolution Rate65%70–75%85%+
Avg. Resolution Time8–12 min2–4 min<2 min
CSAT Score3.8/5.04.2/5.04.6/5.0
Escalation Rate20–25%10–15%<10%
Cost per Issue$8–12$0.15–0.40$0.08–0.20
Avg. Turns per Conv.4–62–31–2

AI agents outperform humans on cost and speed but need tuning to match CSAT. This is where metrics drive improvement.

Building an analytics dashboard​

Aggregate metrics into meaningful dashboards:

from datetime import timedelta
from collections import defaultdict
import json

class AnalyticsDashboard:
    """Aggregated view of support metrics."""
    
    def __init__(self, metrics_list: list[SupportMetric]):
        self.metrics = metrics_list
        self.time_window = None  # Set to filter by date range
    
    def get_summary_statistics(self, time_days: int = 7) -> dict:
        """Get summary stats for the past N days."""
        cutoff = datetime.now() - timedelta(days=time_days)
        recent = [m for m in self.metrics if m.timestamp >= cutoff]
        
        resolution_times = [m.value for m in recent if m.metric_name == "resolution_time"]
        csat_scores = [m.value for m in recent if m.metric_name == "csat"]
        escalations = [m.value for m in recent if m.metric_name == "escalation"]
        costs = [m.value for m in recent if m.metric_name == "cost"]
        
        def safe_avg(values):
            return sum(values) / len(values) if values else 0
        
        def safe_pct(values):
            return (sum(values) / len(values) * 100) if values else 0
        
        return {
            "period_days": time_days,
            "conversations": len(set(m.conversation_id for m in recent)),
            "resolution_time_avg_minutes": round(safe_avg(resolution_times), 1),
            "resolution_time_p95_minutes": round(sorted(resolution_times)[int(len(resolution_times) * 0.95)] if resolution_times else 0, 1),
            "csat_avg": round(safe_avg(csat_scores), 2),
            "csat_detractors": f"{100 - safe_pct([1 for c in csat_scores if c <= 2])}%",  # CSAT 1–2
            "escalation_rate": f"{safe_pct(escalations):.1f}%",
            "cost_per_conversation": f"${round(safe_avg(costs), 2)}",
            "total_cost": f"${round(sum(costs), 2)}",
        }
    
    def get_metrics_by_intent(self, time_days: int = 7) -> dict:
        """Performance breakdown by detected intent."""
        cutoff = datetime.now() - timedelta(days=time_days)
        recent = [m for m in self.metrics if m.timestamp >= cutoff]
        
        by_intent = defaultdict(list)
        for m in recent:
            if m.dimension and "intent" in m.dimension:
                by_intent[m.dimension["intent"]].append(m)
        
        results = {}
        for intent, metrics_for_intent in by_intent.items():
            escalations = [m.value for m in metrics_for_intent if m.metric_name == "escalation"]
            csat = [m.value for m in metrics_for_intent if m.metric_name == "csat"]
            
            results[intent] = {
                "conversations": len(set(m.conversation_id for m in metrics_for_intent)),
                "escalation_rate": f"{(sum(escalations) / len(escalations) * 100):.1f}%" if escalations else "N/A",
                "avg_csat": round(sum(csat) / len(csat), 2) if csat else "N/A",
            }
        
        return results
    
    def get_quality_report(self) -> dict:
        """Identify quality issues for improvement."""
        stats = self.get_summary_statistics()
        
        issues = []
        
        # Check CSAT
        csat = float(stats["csat_avg"])
        if csat < 4.0:
            issues.append({
                "area": "CSAT",
                "severity": "high",
                "current": csat,
                "target": 4.2,
                "recommendation": "Review low-CSAT conversations; improve tone or accuracy"
            })
        
        # Check escalation rate
        escalation_pct = float(stats["escalation_rate"].rstrip("%"))
        if escalation_pct > 15:
            issues.append({
                "area": "Escalation",
                "severity": "medium",
                "current": escalation_pct,
                "target": 10,
                "recommendation": "Improve agent confidence; expand tool capabilities"
            })
        
        # Check cost
        cost = float(stats["cost_per_conversation"].lstrip("$"))
        if cost > 0.40:
            issues.append({
                "area": "Cost",
                "severity": "low",
                "current": cost,
                "target": 0.25,
                "recommendation": "Optimize prompt; use faster model or fewer tool calls"
            })
        
        return {
            "summary": stats,
            "issues": issues,
            "last_updated": datetime.now().isoformat()
        }

# Example
collector = MetricsCollector()
# (assume metrics were recorded)
dashboard = AnalyticsDashboard(collector.metrics)
report = dashboard.get_quality_report()
print(json.dumps(report, indent=2))

A/B testing and improvement loops​

Metrics are useless without experimentation. Use A/B testing to improve:

class ABTester:
    """Run A/B tests on agent improvements."""
    
    def __init__(self, variant_a_prompt: str, variant_b_prompt: str):
        self.variant_a_prompt = variant_a_prompt
        self.variant_b_prompt = variant_b_prompt
        self.variant_a_metrics = []
        self.variant_b_metrics = []
    
    def run_experiment(
        self,
        test_conversations: list[dict],
        metric_name: str
    ) -> dict:
        """Run A/B test on a set of conversations."""
        
        # Split conversations 50/50
        half = len(test_conversations) // 2
        variant_a_convs = test_conversations[:half]
        variant_b_convs = test_conversations[half:]
        
        # Simulate running agent with each variant
        # In production: actually run against live traffic
        variant_a_results = self._evaluate_variant(
            variant_a_convs,
            self.variant_a_prompt
        )
        variant_b_results = self._evaluate_variant(
            variant_b_convs,
            self.variant_b_prompt
        )
        
        # Compare
        metric_a = [r.get(metric_name) for r in variant_a_results if metric_name in r]
        metric_b = [r.get(metric_name) for r in variant_b_results if metric_name in r]
        
        avg_a = sum(metric_a) / len(metric_a) if metric_a else 0
        avg_b = sum(metric_b) / len(metric_b) if metric_b else 0
        improvement = ((avg_b - avg_a) / avg_a * 100) if avg_a > 0 else 0
        
        # Statistical significance (simplified)
        is_significant = abs(improvement) > 5  # >5% improvement
        
        return {
            "metric": metric_name,
            "variant_a_avg": round(avg_a, 2),
            "variant_b_avg": round(avg_b, 2),
            "improvement_percent": round(improvement, 1),
            "is_significant": is_significant,
            "recommendation": "Deploy variant B" if is_significant and improvement > 0 else "Keep variant A",
            "sample_size": len(variant_a_convs) + len(variant_b_convs)
        }
    
    def _evaluate_variant(self, conversations: list[dict], prompt: str) -> list[dict]:
        """Evaluate a variant on a set of conversations (mock)."""
        # In production: run actual inference with the prompt
        # Return metrics for each conversation
        return [
            {
                "csat": 4.2,
                "resolution_time": 2.5,
                "escalation": False
            } for _ in conversations
        ]

# Example
tester = ABTester(
    variant_a_prompt="Original prompt",
    variant_b_prompt="Improved prompt with empathy"
)

result = tester.run_experiment(
    test_conversations=[{} for _ in range(100)],
    metric_name="csat"
)

print(json.dumps(result, indent=2))
# Output: improvement of 8%, significant → deploy variant B

Continuous monitoring and alerting​

Set up monitoring to catch regressions:

class MetricsMonitor:
    """Watch for metric regressions."""
    
    def __init__(self):
        self.baseline = {
            "csat": 4.2,
            "escalation_rate": 0.12,
            "resolution_time": 3.0,
            "cost": 0.25
        }
        self.alert_threshold = 0.10  # Alert if >10% deviation
    
    def check_current_metrics(self, current: dict) -> list[dict]:
        """Check for regressions vs baseline."""
        alerts = []
        
        for metric_name, baseline_value in self.baseline.items():
            current_value = current.get(metric_name)
            if current_value is None:
                continue
            
            deviation = abs(current_value - baseline_value) / baseline_value
            
            if deviation > self.alert_threshold:
                alerts.append({
                    "metric": metric_name,
                    "baseline": baseline_value,
                    "current": current_value,
                    "deviation_percent": round(deviation * 100, 1),
                    "severity": "critical" if deviation > 0.25 else "warning"
                })
        
        return alerts

# Usage
monitor = MetricsMonitor()
current = {
    "csat": 3.9,  # Down from 4.2
    "escalation_rate": 0.18,  # Up from 0.12
    "resolution_time": 3.0,  # Stable
    "cost": 0.30  # Up from 0.25
}

alerts = monitor.check_current_metrics(current)
for alert in alerts:
    print(f"ALERT {alert['severity']}: {alert['metric']} deviated {alert['deviation_percent']}%")

Key Takeaways​

  • Measure five core metrics — resolution time, CSAT, escalation rate, cost per conversation, and conversation turns. These drive everything.
  • Set realistic targets — resolution rate 70–75%, CSAT 4.2+, escalation <15%, cost $0.15–0.40 per conversation.
  • Build aggregated dashboards — summary stats, breakdowns by intent/language/tier, and quality reports that highlight areas to improve.
  • A/B test improvements continuously — run experiments (new prompts, tools, routing rules) against 50% of traffic; deploy if >5% improvement and statistically significant.
  • Monitor for regressions — set baselines, alert if any metric deviates >10%, investigate immediately to prevent customer impact.

Frequently Asked Questions​

How often should I collect CSAT surveys?​

After every conversation, but make it optional (don't force it). Aim for 30–50% response rate. Weight recent responses (last 7 days) more heavily in your average; older data has less signal.

Should I measure individual agent or whole system?​

Measure the whole system first. Then, once you understand system-level trends, break down by intent, language, customer tier, time-of-day, etc. This gives you levers to pull for improvement.

What if my metrics show the agent is worse than humans?​

Investigate why: (1) Wrong tasks? (2) Undertrained or bad prompts? (3) Insufficient tools? (4) Measuring the wrong things? Give the agent 4–12 weeks of iteration before giving up. Most AI agents underperform at launch but improve rapidly with feedback.

How do I report metrics to executives?​

Focus on business impact: cost savings ($X per ticket), volume handled (Y% of tickets, humans free for complex issues), and customer impact (CSAT +0.3 points). Skip technical metrics (tokens, latency) unless they directly drive the business ones.

Can I use metrics to improve the model's prompts automatically?​

Yes, but carefully. Use metrics to identify which intents/scenarios perform worst. Then manually improve the prompts for those scenarios. Avoid auto-prompting (using metrics to generate prompts); it often makes things worse.

Further Reading​