Skip to main content

Self-Consistency: Improving CoT with Multiple Outputs

If one chain of thought is good, are multiple chains of thought better? The answer is a resounding yes. Self-Consistency is a powerful technique that takes Chain-of-Thought prompting to the next level, leveraging the power of diversity to find more reliable answers.

Introduction​

In the previous article, we introduced Chain-of-Thought (CoT) prompting, a method for encouraging LLMs to "think step-by-step." This is a huge leap forward for complex reasoning tasks. However, it has a potential weakness: what if the single chain of thought it generates contains an error? The final answer will be wrong, and we have no way to recover.

This is where Self-Consistency comes in. It's a simple yet incredibly effective technique that builds upon CoT. Instead of generating just one reasoning path, we generate multiple paths and then use a voting mechanism to determine the most consistent final answer. It's like asking a committee of experts to solve a problem independently and then choosing the answer they most agree on.

The Core Idea: Diversity and a "Wisdom of the Crowd" Effect​

Self-Consistency works because there are often many different ways to reason through a problem. By generating several different chains of thought, we explore this diverse space of possible reasoning paths. Even if some of these paths contain errors, it's likely that the majority of them will converge on the correct answer.

This leverages a "wisdom of the crowd" effect within the LLM itself. We are not just taking the first answer the model gives us; we are asking it to reconsider the problem from multiple angles and then aggregating its "opinions."

The key to making this work is to introduce randomness into the generation process. This is typically done by setting the temperature parameter of the LLM to a value greater than 0 (e.g., 0.5 or 0.7). This ensures that when we ask the model the same question multiple times, we get different reasoning paths.

The Self-Consistency Workflow​

The process is straightforward:

  1. Use a CoT Prompt: Start with a good Chain-of-Thought prompt (either Zero-Shot or Few-Shot).
  2. Generate Multiple Completions: Run the same prompt through the LLM multiple times (e.g., 3, 5, or more times). Make sure the temperature is set to a value > 0 to ensure diversity in the outputs.
  3. Extract the Final Answer from Each Completion: Parse each of the generated chains of thought to find the final answer at the end of each one.
  4. Aggregate and Choose the Best Answer: Tally up the final answers. The one that appears most frequently is your final, self-consistent answer.

A Practical Example​

Let's revisit our movie theater problem.

Prompt:

Q: A group of 5 friends is going to the movies. They each buy a ticket for $12 and a popcorn for $8. They also share 2 large sodas that cost $6 each. What was the total cost?

A: Let's think step by step.

Now, let's imagine we run this prompt three times with a temperature of 0.7. We might get the following three completions:

Completion 1:

  1. Cost per friend for ticket and popcorn is $12 + $8 = $20.
  2. Total for 5 friends is 5 * $20 = $100.
  3. Cost for 2 sodas at $6 each is 2 * $6 = $12.
  4. Total cost is $100 + $12 = $112. The answer is 112.

Completion 2 (with a reasoning error):

  1. There are 5 friends.
  2. They each buy a ticket ($12) and popcorn ($8), so that's $20 per person. Total is 5 * $20 = $100.
  3. They share 2 sodas, so each friend pays for 2/5 of a soda. This is getting complicated. Let's try again.
  4. Total for tickets/popcorn is $100.
  5. Total for sodas is 2 * $6 = $12.
  6. The grand total is $100 + $12 = $112. The answer is 112.

Completion 3 (with a different reasoning path):

  1. First, calculate the total cost of tickets: 5 * $12 = $60.
  2. Next, calculate the total cost of popcorn: 5 * $8 = $40.
  3. Then, calculate the total cost of sodas: 2 * $6 = $12.
  4. Finally, add them all together: $60 + $40 + $12 = $112. The answer is 112.

Analysis:

  • We parse the final answer from each completion: 112, 112, 112.
  • We tally the results. The answer 112 appears 3 times.
  • Our final, self-consistent answer is 112.

Now, imagine that in one of the completions, the model made a math error and got 102. The final tally would be 112 (2 votes) and 102 (1 vote). We would still choose 112 as the most consistent and likely correct answer.

When to Use Self-Consistency​

Self-Consistency is particularly useful for:

  • High-Stakes Reasoning: When the accuracy of the final answer is critical.
  • Noisy or Ambiguous Problems: When there might be multiple valid interpretations or ways to solve the problem.
  • Production Systems: It's a great way to improve the reliability of automated reasoning systems.

The main drawback is the increased cost and latency, as you are running the same prompt multiple times. You need to find the right balance between the number of completions and your performance requirements. Often, using just 3 or 5 paths provides a significant boost in accuracy.

Key Takeaways​

  • Self-Consistency improves the reliability of CoT prompting.
  • It works by generating multiple diverse reasoning paths and choosing the most frequent answer.
  • To generate diverse paths, you must set the temperature parameter to a value greater than 0.
  • This technique can significantly reduce errors in tasks that require arithmetic or complex reasoning.
  • The main trade-off is increased cost and latency.

What's Next?​

Self-Consistency explores multiple independent paths to an answer. But what if the reasoning process isn't linear? What if it requires exploring a path, realizing it's a dead end, and then backtracking? In the next article, we will introduce Tree of Thoughts (ToT), a more advanced technique that allows the model to actively explore a tree of reasoning possibilities, making it even more powerful for complex problem-solving.

By embracing Self-Consistency, you are no longer relying on a single train of thought; you are harnessing the collective wisdom of a committee of reasoners to find the most robust and reliable answer.