- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors argue that scaling test-time compute has three main types of error bias: (1) repeated sampling can reproduce a wrong majority answer, (2) longer reasoning can preserve an initially incorrect prediction, and (3) self-evaluation systematically favors later and more verbose thinking. Based on these, this paper presents DIA, a pipeline that uses disagreement as the trigger to choose when to obtain external evidence.
Critical strengths
-
The writing is clear and Figure 1 does a great job of illustrating the main three model-spefic error biases, though they were mainly built on top of existing work.
-
The main contribution of this work is, in my opinion, putting together the main challenges and their solutions for scaling test-time compute in a clean and organized way. The method itself is simple and intuitive.
Critical weaknesses
-
Since scaling test-time compute is the main theme, I would expect compute-matched baselines like self-consistency [1], Best-of-N [2], simple test-time scaling (s1 [3]). It is also important to match the compute budget.
-
Adaptive reasoning should also include Adaptive Self-Consistency [4], which adaptively decides how many samples to use for each query based on a simple beta distribution.
-
The margin is small across datasets and, on average, only outperforms the best baseline by less than 1.5%.
References:
Questions for authors
-
I'm not sure "More compute from the same model is therefore not more evidence" is an accurate description. I understand the points about "model-specific error bias persists across generation, reasoning, and self-evaluation", but this only tells us "more compute does not resolve model-specific error bias" not "more compute =/= more evidence". What does evidence mean here?
-
Also, more compute does, in general, improve the overall accuracy, so the expected error goes down with increased compute. How would the authors frame that? In lines 156 and 157, the Takeaway says: "Increasing the sampling budget of the same model may repeatedly reinforce an existing error mode rather than introduce independent evidence." But on average, test-time scaling methods like self-consistency [1], Best-of-N [2], simple test-time scaling (s1 [3]) does reduce errors.
-
There's also work that tries to do finer refinement and identify error steps to enable better scaling [4], I am curious about what the main distinction is at a conceptual level, and what the performance comparison will look like.
References: