Skip to main content

Confidence Scoring in Document AI: Why It Matters

Confidence scoring is the practice of assigning a numerical score (typically 0.0 to 1.0) to each extracted field or document that represents how confident the model is in its extraction. Without confidence scoring, you treat all extractions equally: a high-quality, legible invoice gets the same weight as a blurry, damaged one. Confidence scoring helps you route low-confidence extractions to human review, improving both accuracy and system reliability. In production systems processing thousands of documents, this becomes critical: do you process an invoice with 95% confidence directly into accounting? Yes. Do you auto-process one with 60% confidence? Absolutely not.

I've seen confidence scoring reduce downstream errors by 70% because it creates a quality gate: high-confidence extractions flow straight to systems; low-confidence ones get human review before integration.

Why Confidence Matters​

The Real-World Cost of Uncertainty​

Imagine processing 10,000 invoices. If you assume 90% accuracy without confidence scoring, approximately 1,000 invoices are wrong. Some of those errors (wrong vendor, wrong amount) cascade downstream: incorrect payments, audit failures, reconciliation nightmares.

With confidence scoring and a 85% confidence threshold, you might auto-process 7,000 invoices (all above the threshold, high accuracy). The remaining 3,000 get quick human review: 5-10 seconds per invoice, total ~1 hour. This dramatically reduces error propagation.

Confidence Scoring Techniques​

Technique 1: Model-Provided Confidence Scores​

Some models report confidence implicitly through logits or token probabilities. Claude doesn't expose logits, but you can ask it to report confidence explicitly:

import anthropic
import base64
import json
from pathlib import Path

def extract_with_explicit_confidence(image_path: str) -> dict:
    """
    Extract data and ask the model to report confidence per field.
    """
    client = anthropic.Anthropic()
    
    image_data = Path(image_path).read_bytes()
    base64_image = base64.standard_b64encode(image_data).decode("utf-8")
    
    confidence_prompt = """Extract invoice data and report confidence per field.

Return JSON:
{
  "fields": {
    "invoice_number": {
      "value": "extracted value",
      "confidence": 0.0-1.0,
      "reasoning": "why you're confident or not"
    },
    "vendor_name": {
      "value": "...",
      "confidence": 0.0-1.0,
      "reasoning": "..."
    },
    "total_amount": {
      "value": "...",
      "confidence": 0.0-1.0,
      "reasoning": "..."
    }
  }
}

Confidence scale:
- 1.0: Field is clearly visible, unambiguous
- 0.8+: Field is visible but slightly unclear (e.g., slight blur)
- 0.6-0.8: Field is present but somewhat ambiguous (e.g., handwritten, faint)
- 0.4-0.6: Field is hard to read but extractable
- 0.0-0.4: Field is very unclear, illegible, or possibly missing
- 0.0: Field is missing entirely

Base your confidence on:
1. Image quality (clarity, contrast, rotation)
2. Field visibility (prominent vs. hidden, large vs. small font)
3. Ambiguity (multiple interpretations possible?)
4. Data completeness (field has full value or partial?)"""
    
    response = client.messages.create(
        model="claude-3-5-sonnet-20241022",
        max_tokens=1024,
        messages=[
            {
                "role": "user",
                "content": [
                    {
                        "type": "image",
                        "source": {
                            "type": "base64",
                            "media_type": "image/jpeg",
                            "data": base64_image
                        }
                    },
                    {
                        "type": "text",
                        "text": confidence_prompt
                    }
                ]
            }
        ]
    )
    
    return json.loads(response.content[0].text)

# Example usage
result = extract_with_explicit_confidence("invoice.jpg")
for field_name, field_data in result["fields"].items():
    print(f"{field_name}: {field_data['value']} "
          f"(confidence: {field_data['confidence']:.1%})")
    print(f"  Reasoning: {field_data['reasoning']}\n")

Technique 2: Image Quality Assessment​

Image quality directly affects extraction accuracy. Assess it before extraction:

from PIL import Image
import numpy as np

def assess_image_quality(image_path: str) -> dict:
    """
    Assess document image quality.
    Returns a quality score and diagnostics.
    """
    img = Image.open(image_path)
    img_array = np.array(img)
    
    # Convert to grayscale if color
    if len(img_array.shape) == 3:
        img_gray = np.mean(img_array, axis=2).astype(np.uint8)
    else:
        img_gray = img_array
    
    diagnostics = {}
    
    # 1. Contrast (standard deviation of pixel intensities)
    contrast = np.std(img_gray)
    diagnostics["contrast"] = float(contrast)
    
    # 2. Brightness (mean pixel intensity)
    brightness = np.mean(img_gray)
    diagnostics["brightness"] = float(brightness)
    
    # 3. Sharpness (edge detection via Laplacian)
    from scipy import ndimage
    laplacian = ndimage.laplace(img_gray)
    sharpness = np.var(laplacian)
    diagnostics["sharpness"] = float(sharpness)
    
    # 4. Resolution
    diagnostics["resolution"] = img.size  # (width, height)
    diagnostics["dpi"] = img.info.get("dpi", (72, 72)) if hasattr(img, "info") else (72, 72)
    
    # Compute quality score (0.0-1.0)
    # Higher contrast, adequate brightness, higher sharpness = higher quality
    quality_score = 0.5
    
    if 50 < contrast < 150:  # Optimal contrast range
        quality_score += 0.25
    elif contrast > 0:
        quality_score += 0.10  # Suboptimal contrast
    
    if 50 < brightness < 200:  # Adequate brightness
        quality_score += 0.15
    
    if sharpness > 100:  # Sharp image
        quality_score += 0.15
    
    # Penalize very low resolution
    width, height = img.size
    if width < 200 or height < 200:
        quality_score -= 0.2
    
    quality_score = max(0.0, min(1.0, quality_score))
    
    return {
        "quality_score": quality_score,
        "diagnostics": diagnostics,
        "quality_level": (
            "excellent" if quality_score > 0.85
            else "good" if quality_score > 0.7
            else "fair" if quality_score > 0.5
            else "poor"
        )
    }

# Example usage
quality = assess_image_quality("invoice.jpg")
print(f"Quality: {quality['quality_level']}")
print(f"Quality score: {quality['quality_score']:.2%}")
print(f"Contrast: {quality['diagnostics']['contrast']:.1f}")
print(f"Sharpness: {quality['diagnostics']['sharpness']:.1f}")

Technique 3: Field Completeness and Validation​

Confidence also depends on whether extracted fields pass validation:

def compute_field_confidence(field_value: str, field_type: str, validation_passed: bool) -> float:
    """
    Compute confidence for a single field based on value completeness and validation.
    """
    base_confidence = 0.5
    
    # Completeness: is the field fully populated?
    if field_value and len(str(field_value).strip()) > 0:
        base_confidence += 0.2
    
    # Validation: does it pass type/format checks?
    if validation_passed:
        base_confidence += 0.3
    else:
        base_confidence -= 0.3
    
    return max(0.0, min(1.0, base_confidence))

def compute_document_confidence(extracted_data: dict, schema: dict) -> dict:
    """
    Compute overall confidence for an extracted document.
    """
    field_confidences = {}
    
    for field_name, field_value in extracted_data.items():
        field_schema = schema["properties"].get(field_name, {})
        field_type = field_schema.get("type", "string")
        
        # Simple validation: check if required and non-null
        is_required = field_name in schema.get("required", [])
        validation_passed = (
            (field_value is not None) if is_required
            else True
        )
        
        confidence = compute_field_confidence(field_value, field_type, validation_passed)
        field_confidences[field_name] = confidence
    
    # Overall confidence: average of field confidences, weighted by required fields
    required_fields = schema.get("required", [])
    required_confidences = [
        field_confidences[f] for f in required_fields
        if f in field_confidences
    ]
    
    if required_confidences:
        overall_confidence = np.mean(required_confidences)
    else:
        overall_confidence = np.mean(list(field_confidences.values()))
    
    return {
        "overall_confidence": float(overall_confidence),
        "field_confidences": field_confidences,
        "requires_review": overall_confidence < 0.80
    }

# Example usage
schema = {
    "type": "object",
    "properties": {
        "invoice_number": {"type": "string"},
        "total_amount": {"type": "number"},
        "vendor_name": {"type": "string"}
    },
    "required": ["invoice_number", "total_amount", "vendor_name"]
}

extracted_data = {
    "invoice_number": "INV-2026-00123",
    "total_amount": 1234.56,
    "vendor_name": "ACME Corp"
}

confidence_result = compute_document_confidence(extracted_data, schema)
print(f"Overall confidence: {confidence_result['overall_confidence']:.2%}")
print(f"Requires review: {confidence_result['requires_review']}")
for field, conf in confidence_result["field_confidences"].items():
    print(f"  {field}: {conf:.2%}")

Quality Gating with Confidence Thresholds​

Use confidence scores to route documents:

def process_with_quality_gate(image_path: str, high_confidence_threshold: float = 0.85, 
                              low_confidence_threshold: float = 0.60) -> dict:
    """
    Process a document with quality gates based on confidence.
    """
    # Extract with confidence
    extracted = extract_with_explicit_confidence(image_path)
    
    # Assess image quality
    img_quality = assess_image_quality(image_path)
    
    # Compute overall confidence
    document_confidence = compute_document_confidence(extracted["fields"], invoice_schema)
    
    # Combine confidence scores
    combined_confidence = (
        document_confidence["overall_confidence"] * 0.6 +
        img_quality["quality_score"] * 0.4
    )
    
    # Route based on confidence
    if combined_confidence >= high_confidence_threshold:
        route = "AUTO_PROCESS"
        action = "Process directly into accounting system"
    elif combined_confidence >= low_confidence_threshold:
        route = "HUMAN_REVIEW"
        action = "Route to quick human review"
    else:
        route = "REJECT"
        action = "Reject and request document rescan"
    
    return {
        "extracted_data": extracted,
        "image_quality": img_quality,
        "document_confidence": document_confidence,
        "combined_confidence": combined_confidence,
        "route": route,
        "action": action,
        "requires_review": combined_confidence < high_confidence_threshold
    }

Tracking Confidence Over Time​

Monitor confidence metrics to improve your system:

def log_extraction_result(document_id: str, confidence: float, 
                          human_reviewed: bool, human_correction_made: bool):
    """
    Log extraction result with confidence for metrics tracking.
    """
    return {
        "document_id": document_id,
        "confidence": confidence,
        "human_reviewed": human_reviewed,
        "human_correction_made": human_correction_made,
        "timestamp": datetime.now().isoformat()
    }

def compute_confidence_metrics(logs: list[dict]) -> dict:
    """
    Compute aggregate metrics from extraction logs.
    """
    import numpy as np
    from datetime import datetime, timedelta
    
    df_logs = [log for log in logs]  # In production, use pandas
    
    high_conf = [log for log in df_logs if log["confidence"] >= 0.85]
    low_conf = [log for log in df_logs if log["confidence"] < 0.60]
    
    high_conf_error_rate = sum(
        log["human_correction_made"] for log in high_conf
    ) / len(high_conf) if high_conf else 0
    
    low_conf_review_rate = sum(
        log["human_reviewed"] for log in low_conf
    ) / len(low_conf) if low_conf else 0
    
    return {
        "avg_confidence": np.mean([log["confidence"] for log in df_logs]),
        "high_confidence_error_rate": high_conf_error_rate,
        "low_confidence_review_rate": low_conf_review_rate,
        "total_documents_processed": len(df_logs),
        "auto_processed_count": sum(1 for log in df_logs if log["confidence"] >= 0.85),
        "human_reviewed_count": sum(1 for log in df_logs if log["human_reviewed"])
    }

Key Takeaways​

  • Confidence scoring assigns a reliability score to each extraction, enabling quality gates and human review routing.
  • Three techniques: model-provided confidence (ask the model explicitly), image quality assessment (contrast, sharpness, resolution), field validation (do values pass type checks?).
  • Combine multiple confidence signals (model confidence, image quality, field validation) for robust quality assessment.
  • Set thresholds: auto-process high-confidence extractions, route medium-confidence to human review, reject low-confidence.
  • Track confidence metrics over time to identify systematic issues and improve extraction quality.

Frequently Asked Questions​

What confidence threshold should I use?​

It depends on your downstream system's tolerance for errors. Finance systems (processing payments) should use high thresholds (0.85-0.95). Informational systems (populating search indexes) can use lower thresholds (0.60-0.70). Start with 0.80 and adjust based on your error rate tolerance.

Can I improve confidence by preprocessing images?​

Absolutely. Deskew, enhance contrast, and clean up scans before extraction. This can boost confidence by 10-20 percentage points because the image is clearer. If image quality is poor, tell users to rescan rather than trying to extract from a bad image.

How do I handle documents that confidently extract wrong data?​

This happens (overconfident models). Track which documents have high confidence but human correction, and flag this for prompt refinement. Add more specific instructions to your extraction prompt to disambiguate confusing cases.

Should I use machine learning to predict confidence?​

For very high-volume systems (100,000+ documents), training a confidence predictor (meta-model) can be worthwhile. Otherwise, rule-based confidence (quality + validation + explicit model scoring) is simpler and more interpretable.

How do I explain confidence to users?​

Be transparent: "This extraction is 92% confident because the image is clear, all required fields are present, and they pass validation checks. A human reviewer would likely approve it without changes." Users appreciate understanding why something is routed to review.

Further Reading​