Skip to main content

Prompt testing and regression detection

Prompt regression testing detects when a prompt change inadvertently breaks functionality or degrades quality for cases that previously worked well. Unlike traditional software, where you compare outputs bit-by-bit against expectations, prompt regression testing uses statistical and semantic comparisons: you run two prompt variants on the same test set, measure differences in outputs (accuracy, tone, adherence to constraints), and flag unexpected regressions. A regression might be a drop in accuracy, increased hallucinations, changed formatting, or a shift in tone—anything that violates the expected behavior profile you built with the baseline prompt.

Why Prompt Regression Testing Matters​

Prompt changes are easy to make but hard to understand their full impact. A small rewording that improves performance on one use case might hurt performance on another. You might remove a constraint to reduce token usage, only to discover days later that the model now generates unsafe outputs. Without regression testing, you rely on manual spot-checking, which catches only obvious failures. Regression testing automates this: it compares every new prompt against a baseline on a diverse test set, flags unexpected changes, and gives you confidence that your iteration improved the system overall.

Setting Up Baseline and Variant Prompts​

Store prompts in version control as plain-text files, each with a version tag. A baseline prompt is a known-good prompt that currently runs in production. Variant prompts are candidates you are testing.

# prompts/system-instruction.v1.txt
You are a helpful customer support assistant. Answer questions about product returns, shipping, and refunds.
- Keep responses under 150 words.
- Be empathetic and professional.
- If you do not know the answer, say so and suggest contacting [email protected].

# prompts/system-instruction.v2.txt (variant with shorter preamble)
You are a customer support assistant. Help with returns, shipping, refunds. Keep answers under 150 words. If unsure, suggest [email protected].

For each variant, you will run inference on a test set and compare outputs. The test set should include real customer queries that represent your production traffic distribution.

Implementing Regression Detection​

Create a regression test script that compares outputs from baseline and variant prompts on the same test cases.

import json
from anthropic import Anthropic
from sentence_transformers import SentenceTransformer
import difflib

client = Anthropic()
embedder = SentenceTransformer("all-MiniLM-L6-v2")

def compare_prompts(baseline_prompt: str, variant_prompt: str, test_dataset: str) -> dict:
    """Compare outputs of two prompt variants on a test set."""
    with open(test_dataset) as f:
        tests = [json.loads(line) for line in f]
    
    baseline_outputs = []
    variant_outputs = []
    
    # Run baseline
    for test in tests:
        response = client.messages.create(
            model="claude-3-5-sonnet-20241022",
            max_tokens=500,
            system=baseline_prompt,
            messages=[{"role": "user", "content": test["query"]}]
        )
        baseline_outputs.append(response.content[0].text)
    
    # Run variant
    for test in tests:
        response = client.messages.create(
            model="claude-3-5-sonnet-20241022",
            max_tokens=500,
            system=variant_prompt,
            messages=[{"role": "user", "content": test["query"]}]
        )
        variant_outputs.append(response.content[0].text)
    
    # Compare outputs
    regressions = []
    for i, (test, baseline, variant) in enumerate(
        zip(tests, baseline_outputs, variant_outputs)
    ):
        # Compute semantic similarity
        baseline_emb = embedder.encode(baseline)
        variant_emb = embedder.encode(variant)
        similarity = float(
            (baseline_emb @ variant_emb) / (1 + abs(len(baseline) - len(variant)) / 100)
        )
        
        # Flag if outputs diverge significantly
        if similarity < 0.75:  # threshold
            regressions.append({
                "query": test["query"],
                "category": test.get("category"),
                "baseline": baseline,
                "variant": variant,
                "similarity": similarity,
                "diff": "\n".join(
                    difflib.unified_diff(
                        baseline.split("\n"), 
                        variant.split("\n"),
                        lineterm=""
                    )
                )
            })
    
    return {
        "total_tests": len(tests),
        "regressions_found": len(regressions),
        "regression_rate": len(regressions) / len(tests),
        "regressions": regressions,
        "pass": len(regressions) == 0  # gate fails if ANY regressions found
    }

if __name__ == "__main__":
    import sys
    result = compare_prompts(
        baseline_prompt=open("prompts/system-instruction.v1.txt").read(),
        variant_prompt=open("prompts/system-instruction.v2.txt").read(),
        test_dataset="test_data/customer_queries.jsonl"
    )
    print(json.dumps(result, indent=2))
    sys.exit(0 if result["pass"] else 1)

Semantic Drift Detection​

Not all changes are regressions. A paraphrase that says the same thing differently should not be flagged as a regression. Use semantic drift detection to identify meaningful divergences: compare outputs not just for similarity but for consistency on specific dimensions (tone, answer correctness, constraint adherence).

def detect_semantic_drift(
    baseline_output: str,
    variant_output: str,
    constraints: dict
) -> dict:
    """Detect whether variant violates expected constraints."""
    issues = []
    
    # Check length constraint
    if constraints.get("max_tokens"):
        max_len = constraints["max_tokens"] * 4  # rough estimate
        if len(variant_output) > max_len:
            issues.append("exceeds_length_constraint")
    
    # Check tone constraint (simple heuristic: presence of exclamation marks)
    if constraints.get("tone") == "professional":
        if variant_output.count("!") > baseline_output.count("!") + 2:
            issues.append("tone_drift_too_enthusiastic")
    
    # Check format constraint
    if constraints.get("format") == "json":
        try:
            json.loads(variant_output)
        except json.JSONDecodeError:
            issues.append("format_not_json")
    
    # Semantic similarity as catch-all
    baseline_emb = embedder.encode(baseline_output)
    variant_emb = embedder.encode(variant_output)
    similarity = float(baseline_emb @ variant_emb)
    
    if similarity < 0.7:
        issues.append("semantic_divergence")
    
    return {
        "is_drift": len(issues) > 0,
        "issues": issues,
        "similarity": similarity
    }

Regression Severity Levels​

Not all regressions block deployment. Categorize regressions by severity:

  • Critical: Output fails a safety constraint (generates harmful content, violates format, exposes secrets). Block deployment.
  • High: Accuracy drops, answer is factually incorrect, or constraint is violated. Require manual review.
  • Medium: Output tone shifts, length changes, or semantic similarity is 0.7-0.8. Flag for inspection but allow override.
  • Low: Minor phrasing changes with similarity 0.8+. Allow deployment but log for observability.
def classify_regression_severity(
    baseline: str,
    variant: str,
    similarity: float,
    safety_check_pass: bool
) -> str:
    """Classify regression severity: critical, high, medium, low, or none."""
    if not safety_check_pass:
        return "critical"
    if similarity < 0.65:
        return "high"
    if similarity < 0.75:
        return "medium"
    if similarity < 0.85:
        return "low"
    return "none"

Continuous Regression Monitoring​

As you deploy prompt updates to production, track regressions over time. Log the similarity score for every inference and compute a weekly regression rate. If the rate spikes (e.g., 10% of outputs diverging significantly from baseline), alert the team and investigate the latest prompt change.

def log_regression_metric(query: str, baseline_sim: float, tags: dict):
    """Log regression metric to observability system."""
    # Send to Datadog, New Relic, or similar
    import time
    metric_name = "llm.prompt.regression_similarity"
    timestamp = int(time.time())
    # Pseudo-code: monitoring_client.gauge(metric_name, baseline_sim, tags=tags, timestamp=timestamp)
    print(f"{metric_name}: {baseline_sim} at {timestamp}")

Key Takeaways​

  • Prompt regression testing compares outputs from baseline and variant prompts on a test set, detecting unexpected changes in behavior.
  • Use semantic similarity and constraint checking to identify regressions without requiring exact-match comparisons.
  • Categorize regressions by severity (critical, high, medium, low) and gate deployment based on severity.
  • Continuously monitor regression rates in production; spike detection triggers investigation of recent prompt changes.
  • Store prompts in version control with clear versioning so you can reproduce any historical behavior.

Frequently Asked Questions​

How often should I run regression tests?​

On every prompt change before merging to main (pre-merge in CI/CD). For production prompts, run a daily baseline comparison to detect unexpected drift. For feature branches, run regressions before asking for code review.

What should I do if a regression is intentional (I meant to change behavior)?​

Add a regression exception file that lists intentional deviations. Document why the change is acceptable (e.g., "changed tone from formal to conversational, expected similarity drop from 0.95 to 0.80"). Review exceptions monthly and remove old ones. This prevents legitimate improvements from being blocked.

How do I handle regressions in only one category (e.g., Spanish queries)?​

Stratify your test set by category and compute regression rates per category. A prompt change might regress Spanish outputs but improve English. Run regression detection per-category and gate on weighted average or per-category thresholds. This gives you finer-grained control.

Can I use model outputs as a baseline instead of a previous prompt version?​

Not recommended. A model's baseline outputs change with fine-tuning and model updates. Use a fixed baseline prompt (versioned in Git) as your reference point. If you update the model, re-establish the baseline and measure regressions against the new model.

How do I avoid regression test fatigue (too many false positives)?​

Tune your similarity threshold based on domain. For technical writing, higher similarity (0.85+) is appropriate. For creative writing, lower similarity (0.70) is acceptable. Monitor regression flags and adjust thresholds monthly. If you log 5+ false positives per day, your threshold is too strict.

Further Reading​