Skip to main content

Fact Verification: Research Agent Accuracy

Fact verification is the quality gate that separates a trustworthy research agent from one that spreads misinformation. Without verification, an agent might synthesize a report citing sources it fabricated, misquote numbers, or claim consensus where disagreement exists. This article teaches you to build multi-layer verification into the agent: checking claims against the sources that allegedly support them, cross-verifying facts across independent sources, and flagging unsupported assertions.

Verification happens in two phases: (1) source verification, checking that cited facts are actually present in the source, and (2) cross-source verification, checking that multiple independent sources support a claim before declaring it established fact. An agent that runs both phases catches 80–90% of hallucinations and misattributions before they reach the final report.

How Do You Detect Hallucinated Citations?​

The most common agent mistake is citing a fact to a source that doesn't actually contain that fact. This happens because LLMs are pattern-matching machines, not databases—they hallucinate plausible citations. To detect this, re-read the source and ask the LLM directly: "Does this source explicitly state this claim?"

from anthropic import Anthropic

client = Anthropic()

def verify_claim_in_source(
    claim: str,
    source_text: str,
    source_url: str
) -> dict:
    """
    Verify that a claimed fact is actually present in the source.
    Returns: {"verified": bool, "quote": str|None, "confidence": float}
    """
    
    system_prompt = """You are a fact-checker. Your job is to verify whether a specific claim
is explicitly supported by a source text.

Rules:
1. Only consider the claim VERIFIED if the exact fact (or equivalent restatement) is present in the text.
2. Do NOT infer, interpret, or extrapolate. Require explicit evidence.
3. If the claim is close but slightly different from the source, mark as PARTIAL.
4. Return a JSON object with your verdict.

Output format:
{
  "verified": true|false,
  "verdict": "VERIFIED|PARTIAL|NOT FOUND|CONTRADICTED",
  "quote": "Exact quote from source (if verified), else null",
  "reasoning": "Brief explanation of your decision",
  "confidence": 0.0 to 1.0
}"""

    prompt = f"""Claim to verify: "{claim}"

Source URL: {source_url}

Source text:
{source_text}

Is this claim explicitly supported by the source text?"""

    response = client.messages.create(
        model="claude-opus-4-1",
        max_tokens=500,
        system=system_prompt,
        messages=[{"role": "user", "content": prompt}]
    )
    
    import json
    text = response.content[0].text
    try:
        return json.loads(text)
    except:
        start = text.find('{')
        end = text.rfind('}') + 1
        return json.loads(text[start:end])

# Example
claim = "TSMC achieved 0.65% yield on 1.4nm in May 2026"
source_text = "TSMC announced 0.65% yields on 1.4nm process. CEO Wei said production ramping."
result = verify_claim_in_source(
    claim,
    source_text,
    "https://example.com/tsmc-news"
)
print(result)
# Output: {"verified": true, "verdict": "VERIFIED", "confidence": 0.95, ...}

Cross-Verification: Requiring Consensus Across Independent Sources​

A single source is never enough. A robust agent requires corroboration: if a fact appears in only one source, flag it as "unverified but cited" in the report. If it appears in 2+ independent sources, mark it as "verified". Here's the pattern:

from collections import defaultdict

class FactVerifier:
    def __init__(self, min_sources_for_consensus: int = 2):
        self.min_sources = min_sources_for_consensus
        self.verified_facts = defaultdict(list)  # fact -> list of sources
    
    def register_fact(self, fact: str, source_url: str, verified: bool):
        """Record that a fact was found in a source."""
        if verified:
            self.verified_facts[fact].append(source_url)
    
    def get_consensus_level(self, fact: str) -> dict:
        """
        Determine consensus on a fact across sources.
        Returns: {
            "fact": str,
            "consensus": "established|supported|isolated|contradicted",
            "source_count": int,
            "sources": list[str]
        }
        """
        sources = self.verified_facts.get(fact, [])
        source_count = len(sources)
        
        if source_count >= self.min_sources:
            consensus = "established"
        elif source_count == 1:
            consensus = "isolated"
        else:
            consensus = "unsupported"
        
        return {
            "fact": fact,
            "consensus": consensus,
            "source_count": source_count,
            "sources": sources
        }
    
    def generate_confidence_label(self, fact: str) -> str:
        """Generate a text label for report inclusion."""
        consensus = self.get_consensus_level(fact)["consensus"]
        
        if consensus == "established":
            return "[Established: Verified by 2+ sources]"
        elif consensus == "isolated":
            return "[Unverified: Single source only]"
        else:
            return "[Unconfirmed: No sources found]"

# Example usage in agent loop
verifier = FactVerifier(min_sources_for_consensus=2)

# Simulate processing multiple sources
facts = [
    ("TSMC 0.65% yield", "https://tsmc-official.com/news", True),
    ("TSMC 0.65% yield", "https://semi-industry.com/analysis", True),
    ("Samsung 0.5% yield", "https://samsung-press.com/release", True),
]

for fact, url, verified in facts:
    verifier.register_fact(fact, url, verified)

# Check consensus
tsmc_consensus = verifier.get_consensus_level("TSMC 0.65% yield")
print(f"TSMC fact: {tsmc_consensus['consensus']} ({tsmc_consensus['source_count']} sources)")
# Output: "established (2 sources)"

samsung_consensus = verifier.get_consensus_level("Samsung 0.5% yield")
print(f"Samsung fact: {samsung_consensus['consensus']} ({samsung_consensus['source_count']} sources)")
# Output: "isolated (1 source)"

Detecting and Flagging Contradictions​

When sources disagree, the agent should flag the contradiction explicitly rather than averaging or picking one. This preserves nuance and allows readers to investigate further.

def detect_and_summarize_contradictions(
    facts_by_claim: dict[str, list[dict]]
) -> list[dict]:
    """
    Identify and summarize contradictions in the fact base.
    
    facts_by_claim: {
        "TSMC yield": [
            {"value": "0.65%", "year": 2026, "source": "url1"},
            {"value": "0.58%", "year": 2026, "source": "url2"}
        ]
    }
    """
    contradictions = []
    
    for claim, fact_list in facts_by_claim.items():
        if len(set(f["value"] for f in fact_list)) > 1:
            # Multiple different values for same claim
            contradictions.append({
                "claim": claim,
                "disagreement": fact_list,
                "severity": "high" if all(f["year"] == fact_list[0]["year"] else "medium"
                           for f in fact_list)
            })
    
    return contradictions

# Example
facts_by_claim = {
    "TSMC 1.4nm yield": [
        {"value": "0.65%", "year": 2026, "source": "tsmc-official"},
        {"value": "0.58%", "year": 2026, "source": "analyst-firm"}
    ]
}

contradictions = detect_and_summarize_contradictions(facts_by_claim)
if contradictions:
    for c in contradictions:
        print(f"Contradiction: {c['claim']}")
        for fact in c['disagreement']:
            print(f"  - {fact['value']} (from {fact['source']})")

Building a Verification Report for Transparency​

Include verification status in the final report so readers can judge credibility:

ClaimEvidenceStatusSources
TSMC 1.4nm yield 0.65%Quarterly announcementVerifiedTSMC, Semi Industry News
Samsung competing at 0.5%Single analyst estimateUnverifiedTechCrunch
Intel targeting 0.3% by Q4Rumor from unnamed sourceUnconfirmedVentureBeat

Key Takeaways​

  • Verify each cited claim against the source that allegedly contains it, using explicit source-reading to catch hallucinated citations.
  • Require consensus across 2+ independent sources before marking a fact as "established"; label isolated claims as "unverified but cited".
  • Detect contradictions across sources and report them explicitly rather than averaging—this preserves nuance and invites reader investigation.
  • Include verification status in the final report (verified, unverified, unconfirmed) to maintain transparency about evidence strength.

Frequently Asked Questions​

What if two sources contradict each other?​

Report both perspectives in your findings section. Don't suppress the disagreement. Include both source URLs and let the reader judge. Example: "Source A claims 0.65%, while Source B (analyst estimate) suggests 0.58%."

How many sources do I need to verify a claim?​

2–3 independent sources covering the same time period is the standard. For major claims, aim for 3+. For background facts, 1 primary source (e.g., official documentation) is sufficient.

What if a source says something different from how my agent interpreted it?​

Flag it as a partial or non-match, then move on. Don't try to convince the LLM the source meant something else. Trust the verification step over the agent's paraphrase.

Should I re-verify facts that the agent cited multiple times?​

Yes, verify each citation at least once. If the same fact appears 5 times in the report, verify it only once but note the duplicate citations as confidence boosters.

Further Reading​