- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how to allocate a shared output-token budget among four sampled response streams per question. It compares four decision rules: uniform, random, round-robin, and an agreement rule that stops once two visible answers match. Agreement matches round-robin in accuracy with slightly lower token cost. Random allocation performs best among the four rules, while a post hoc sequential control that completes streams one at a time performs best overall. This shows that longer completed responses being more often wrong does not mean shorter reasoning would help.
Critical strengths
-
All strategies are evaluated on the same pre-sampled trajectories, so accuracy differences reflect the allocation rules rather than sampling noise.
-
Transparent reporting of negative results and limitations, e.g., post hoc changes to numerical normalization and scheduling are clearly disclosed.
Critical weaknesses
- The main takeaway, that the correlation between response length and accuracy does not imply the effect of an intervention, is a well-established methodological principle. The paper offers no new theory or in-depth mechanistic explanation, so its novelty and technical contribution are limited.
- The budget regime is unrealistic. The total budget (2,048/4,096 tokens) is shared by four streams, leaving each stream about 1,024 tokens, often too few to finish a single reasoning trace. As Table 4 shows, uniform, round-robin and agreement have identical no-answer rates, so accuracy is largely determined by whether any stream finishes, not by the allocation rule. In addition, the 4,096-token generation ceiling appears to truncate many responses (Figure 2); incorrect and missing answers should be reported separately.
- The sequential control is the most direct and competitive comparison, yet it is included only as a post hoc control. It should be treated as a primary baseline.
- Experiments use only 300 GSM8K questions and two models, without other reasoning domains, which limits empirical generality.
Questions for authors
- To the best of my knowledge, self-consistency is typically used to scale test-time compute beyond a single trace. What is the motivation for splitting a total budget comparable to a single reasoning trace across four streams? Is there a practical setting where this regime applies?
- Would the conclusions change under a larger total budget (e.g., 4x the typical trace length), where early stopping in self-consistency is usually applied?
2 comments
I sincerely thank the authors for their response. However, my concerns about this work remain, and I believe Reviewers nicZ and ZXEG have raised similar concerns. While I agree that two-answer agreement can save tokens to some extent, its effect on accuracy remains unclear. I will therefore maintain my original score.
Thank you. We agree that sequential completion should be a primary baseline and have run a new longer-budget confirmation experiment. It uses 64 fixed GSM8K questions, eight independent Qwen3-4B-Thinking-2507 samples per question, and an 8192-token cap per sample. All 512 samples terminated with EOS; their mean length is 1728 tokens, so an 8192-token question budget is 4.7 times the mean single-trace length.
With four streams at this budget, fixed-order sequential completion scores 95.3% with zero missing answers and uses 5386 mean tokens. Adding two-answer agreement stopping preserves exactly the same question-level accuracy and zero-missing rate while reducing mean use to 3333 tokens; the paired saving is 2053 tokens [1787, 2315]. At the larger eight-stream, 32768-token setting, both sequential policies score 96.9% with zero missing answers, while agreement reduces mean use from 13476 to 3458 tokens, a paired saving of 10017 [8484, 11650].
We also separated missing from incorrect outcomes. In the four-stream 4096-token round-robin-agreement condition, the 64 questions contain 31 correct, 33 missing, and 0 explicit wrong outcomes. Sequential agreement at the same budget yields 59 correct, 4 missing, and 1 explicit wrong. Across the full 512 completed traces, 95.7% are correct, 4.3% are explicit wrong, and none are missing. These results confirm that the original small-budget comparison primarily measured answer availability. We will make sequential allocation primary, report missing and wrong outcomes separately, and limit the claim to token cost rather than serving latency.