Skip to main content

Self-Consistency: Improve Chain-of-Thought Prompts

Self-Consistency improves Chain-of-Thought (CoT) prompting by generating multiple independent reasoning paths and selecting the answer that appears most frequently across those paths. This "wisdom of crowds" approach reduces errors by 5–20% on complex arithmetic and logic tasks while requiring only a 3–5x increase in API calls.

Key Takeaways​

  • Self-Consistency builds on CoT by generating 3–5 independent reasoning paths instead of one
  • A voting mechanism selects the final answer that appears most frequently, improving accuracy by 15–20% on benchmark tasks
  • Temperature settings > 0 (e.g., 0.7) introduce diversity; without it, the LLM generates identical outputs
  • Trade-off: higher accuracy but 3–5x increased latency and API cost per query
  • Best for high-stakes reasoning tasks where accuracy outweighs latency concerns

How Self-Consistency Works​

Self-Consistency is an elegant extension of Chain-of-Thought prompting that addresses a fundamental weakness: what if the single reasoning path the model generates is incorrect? By generating multiple independent solutions and aggregating them, you leverage the LLM's ability to explore different reasoning strategies.

The key insight is that LLMs contain multiple valid reasoning paths for the same problem. By introducing randomness via temperature sampling, you activate different paths on successive calls. When you aggregate these outputs via majority voting, incorrect paths (statistical outliers) are filtered out.

This is mathematically sound: if an LLM has 80% accuracy per path and you generate 5 independent paths, voting selects the correct answer with P(majority correct) = 98.3% (binomial distribution). Real benchmarks show 5–20% accuracy gains on arithmetic and symbolic reasoning (Wang et al., 2023).

The Self-Consistency Workflow​

Step 1: Prepare a Chain-of-Thought Prompt​

Start with a solid CoT prompt—either zero-shot or few-shot. The prompt itself remains unchanged; only the inference process differs.

Q: A group of 5 friends is going to the movies. They each buy a ticket for $12 and popcorn for $8. They share 2 large sodas at $6 each. What was the total cost?

A: Let's think step by step.

Step 2: Generate Multiple Completions with Temperature > 0​

Run the same prompt multiple times, setting temperature > 0 (e.g., 0.5–0.7) to ensure diversity:

import openai

prompt = """Q: A group of 5 friends is going to the movies. They each buy a ticket for $12 and popcorn for $8. They share 2 large sodas at $6 each. What was the total cost?

A: Let's think step by step."""

client = openai.OpenAI(api_key="your-api-key")
completions = []

for i in range(5):  # Generate 5 independent paths
    response = client.chat.completions.create(
        model="gpt-4",
        messages=[{"role": "user", "content": prompt}],
        temperature=0.7,  # Required: must be > 0 for diversity
        max_tokens=500
    )
    completions.append(response.choices[0].message.content)
    print(f"Path {i+1}:\n{response.choices[0].message.content}\n")

Step 3: Extract Final Answers​

Parse the final answer from each completion. For arithmetic problems, look for the final numeric answer; for multiple-choice, extract the selected option:

def extract_final_answer(completion_text):
    """
    Extract the numeric answer from a completion.
    Looks for patterns like 'The answer is 112' or 'Total: $112'.
    """
    import re
    # Match "answer is [number]" or "total: $[number]"
    match = re.search(r'(?:answer is|total[:\s]+\$?)(\d+)', completion_text, re.IGNORECASE)
    if match:
        return int(match.group(1))
    return None

answers = [extract_final_answer(c) for c in completions]
print(f"Extracted answers: {answers}")
# Output: Extracted answers: [112, 112, 112, 112, 112]

Step 4: Aggregate via Majority Voting​

Select the answer that appears most frequently:

from collections import Counter

def select_by_self_consistency(answers):
    """Vote for the most common answer."""
    counter = Counter(answers)
    most_common_answer, vote_count = counter.most_common(1)[0]
    confidence = vote_count / len(answers)
    return most_common_answer, confidence

final_answer, confidence = select_by_self_consistency(answers)
print(f"Self-Consistent Answer: {final_answer} (Confidence: {confidence:.1%})")
# Output: Self-Consistent Answer: 112 (Confidence: 100.0%)

Real-World Example: Arithmetic Problem​

Suppose the model generates these 5 completions:

Path 1:

  1. Ticket + popcorn per friend: $12 + $8 = $20
  2. Total for 5 friends: 5 × $20 = $100
  3. Sodas: 2 × $6 = $12
  4. Grand total: $100 + $12 = $112 The answer is 112.

Path 2 (minor reasoning detour, correct answer):

  1. There are 5 friends.
  2. Per-friend cost: $12 ticket + $8 popcorn = $20 each.
  3. All tickets + popcorn: 5 × $20 = $100.
  4. Shared sodas: 2 × $6 = $12 (split among 5, but total is still $12). The answer is 112.

Path 3 (alternative approach):

  1. Total tickets: 5 × $12 = $60
  2. Total popcorn: 5 × $8 = $40
  3. Total sodas: 2 × $6 = $12
  4. Sum: $60 + $40 + $12 = $112 The answer is 112.

Path 4 (contains an arithmetic error):

  1. Per-friend ticket + popcorn: $20
  2. 5 friends: 5 × $20 = $100
  3. 2 sodas at $6: 2 × $6 = $10 (error)
  4. Total: $100 + $10 = $110 The answer is 110.

Path 5:

Tickets: 5 × $12 = $60 Popcorn: 5 × $8 = $40 Sodas: $12 Total: $112 The answer is 112.

Voting Result:

  • Answer 112: 4 votes
  • Answer 110: 1 vote
  • Final answer: 112 (80% confidence)

Even though one path contained an error, the majority consensus is correct. Without self-consistency, if the LLM had happened to generate Path 4 on the first try, the answer would be wrong.

When to Use Self-Consistency​

Ideal Use Cases​

  • High-stakes reasoning (medical diagnosis, financial planning, legal analysis)
  • Arithmetic and symbolic reasoning (math word problems, logic puzzles)
  • Multi-step decision-making (planning, code generation, complex explanations)
  • Benchmark-critical applications (AI evaluation, academic research)

When NOT to Use​

  • Real-time, low-latency requirements (customer chat, live translation)
  • Budget-constrained deployments (cost-per-query is critical)
  • Simple classification or retrieval (single-path tasks have no variation)

Cost and Latency Trade-offs​

Generating 5 paths costs 5x the API calls of a single prompt. For OpenAI's GPT-4, that's ~$0.03 per query (vs. $0.006 single-path). The latency increases by ~5x as well—5 sequential API calls take ~5 seconds.

When to pay the cost:

  • Accuracy gain of 10–20% is worth $0.024 per query if the task has high business value
  • Latency of 5 seconds is acceptable for batch processing or offline analysis
  • Volume is moderate (< 1M queries/month) to keep costs manageable

Cost optimization:

# Use fewer paths for lower-stakes tasks
if task_importance == "low":
    num_paths = 3  # 60% of the cost, ~95% of accuracy benefit
elif task_importance == "high":
    num_paths = 7  # 70% accuracy gain typical
else:
    num_paths = 5  # Sweet spot for most applications

Implementation Best Practices​

Setting Temperature Correctly​

Temperature must be > 0 to generate diversity. Common settings:

# Conservative diversity (more coherent, less exploration)
temperature = 0.3

# Moderate diversity (good balance)
temperature = 0.7

# High diversity (explores more reasoning paths, may be less coherent)
temperature = 1.0

For Self-Consistency, 0.5–0.7 is typical. Below 0.3, outputs converge to identical text (negating the technique's benefit). Above 1.0, outputs become too random and reasoning quality drops.

Answer Extraction Robustness​

Real LLM outputs vary in format. Build a robust extractor:

def extract_answer_robust(text):
    """Handle multiple answer formats."""
    import re
    
    patterns = [
        r'(?:answer|result|total)[\s:]*\$?(\d+)',
        r'\*\*(\d+)\*\*',  # Bold formatting
        r'^\d+$',  # Standalone number
        r'(\d+)(?:\s|$)'  # Number followed by space or end
    ]
    
    for pattern in patterns:
        match = re.search(pattern, text, re.IGNORECASE | re.MULTILINE)
        if match:
            return int(match.group(1))
    
    return None

Handling Tie Votes​

If two answers tie, escalate to a tiebreaker:

def select_with_tiebreaker(answers):
    """Select the answer with majority consensus, or use tiebreaker."""
    from collections import Counter
    counter = Counter(answers)
    
    if len(counter) == 0:
        return None
    
    # Get top 2 answers
    top_two = counter.most_common(2)
    
    if len(top_two) == 1 or top_two[0][1] > top_two[1][1]:
        return top_two[0][0]  # Clear majority
    
    # Tie: return the smaller number (conservative bias)
    return min(top_two[0][0], top_two[1][0])

Advanced Variations​

Weighted Voting​

Not all reasoning paths are equally confident. Weight by reasoning coherence:

def weighted_self_consistency(completions_with_confidence):
    """
    Args:
        completions_with_confidence: list of (completion_text, confidence_score)
    """
    from collections import defaultdict
    import re
    
    weighted_votes = defaultdict(float)
    
    for text, confidence in completions_with_confidence:
        answer = extract_final_answer(text)
        if answer is not None:
            weighted_votes[answer] += confidence
    
    if not weighted_votes:
        return None
    
    return max(weighted_votes.items(), key=lambda x: x[1])[0]

Majority Voting with Confidence Threshold​

Require a minimum confidence level:

def consensus_with_threshold(answers, min_confidence=0.6):
    """
    Return the answer only if it meets the confidence threshold.
    Otherwise, escalate for human review.
    """
    from collections import Counter
    counter = Counter(answers)
    
    if not counter:
        return None, 0
    
    most_common_answer, vote_count = counter.most_common(1)[0]
    confidence = vote_count / len(answers)
    
    if confidence >= min_confidence:
        return most_common_answer, confidence
    else:
        return None, confidence  # Flag for escalation

Frequently Asked Questions​

Do I need to set temperature when using Self-Consistency?​

Yes, absolutely. Temperature must be > 0 (typically 0.5–0.7). If temperature = 0, the model produces identical outputs every time, and Self-Consistency provides no benefit. This is the #1 implementation mistake.

How many paths should I generate?​

3 paths give ~60% of the benefit; 5 paths give ~90%; 7+ paths show diminishing returns. Most practitioners use 5 as a sweet spot between accuracy and cost. For mission-critical systems, use 7–10.

Can I combine Self-Consistency with other techniques?​

Yes. Self-Consistency pairs well with few-shot prompting, chain-of-thought, and tree-of-thought. It's orthogonal to these techniques—you generate multiple CoT outputs, not multiple variations of the CoT prompt itself.

What if all 5 paths give different answers?​

This signals low task fit for Self-Consistency—the task may be ambiguous, or the prompt may be under-specified. Clarify the prompt or escalate to human review. A 5-way tie occurs in <2% of well-posed problems.

Does Self-Consistency work with non-numeric answers?​

Yes, but voting is trickier. For multiple-choice, vote on the selected option (A/B/C/D). For free-text, use semantic similarity clustering or embedding-based voting instead of exact string matching.

Further Reading​