acceptodds
Under review as a conference paper at ICLR 2027

From Completed Length to Prefix Decisions: A Paired Budget Audit of Reasoning Models

Abstract

A completed response's length can predict correctness while providing little guidance about when to stop generation. We examine this distinction in a paired prefix-replay study of Qwen3-4B-Thinking-2507 and DeepSeek-R1-Distill-Qwen-7B: four sampled streams for each of 300 GSM8K test questions, totaling 2,400 trajectories. An agreement rule stops once two visible numerical answers match; its threshold is chosen on a separate training-split development set. Under a 4,096-token ceiling, agreement scores 58.0% and 59.3%, exactly matching round-robin's question-level correctness. It consumes 131 and 235 fewer tokens per question, but random allocation scores 67.7% and 67.0%. A post hoc control that finishes streams in fixed order reaches 88.3% on both checkpoints. In contrast, retrospective within-question comparisons show longer incorrect than correct responses on the subsets containing both. The results separate that completed-length association from the effect of a prefix-only decision: how the budget reaches a final answer matters more here than stopping on agreement. We report paired uncertainty, all controls, and separate costs for replay and corpus generation. The measured costs count tokens; online serving latency is not evaluated.

Primary area:
foundation or frontier models, including LLMs
Keywords:
reasoning language models, uncertainty estimation, chain-of-thought length, early stopping, self-consistency, test-time compute

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_mHF1OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to allocate a shared output-token budget among four sampled response streams per question. It compares four decision rules: uniform, random, round-robin, and an agreement rule that stops once two visible answers match. Agreement matches round-robin in accuracy with slightly lower token cost. Random allocation performs best among the four rules, while a post hoc sequential control that completes streams one at a time performs best overall. This shows that longer completed responses being more often wrong does not mean shorter reasoning would help.

Critical strengths
  • All strategies are evaluated on the same pre-sampled trajectories, so accuracy differences reflect the allocation rules rather than sampling noise.

  • Transparent reporting of negative results and limitations, e.g., post hoc changes to numerical normalization and scheduling are clearly disclosed.

Critical weaknesses
  • The main takeaway, that the correlation between response length and accuracy does not imply the effect of an intervention, is a well-established methodological principle. The paper offers no new theory or in-depth mechanistic explanation, so its novelty and technical contribution are limited.
  • The budget regime is unrealistic. The total budget (2,048/4,096 tokens) is shared by four streams, leaving each stream about 1,024 tokens, often too few to finish a single reasoning trace. As Table 4 shows, uniform, round-robin and agreement have identical no-answer rates, so accuracy is largely determined by whether any stream finishes, not by the allocation rule. In addition, the 4,096-token generation ceiling appears to truncate many responses (Figure 2); incorrect and missing answers should be reported separately.
  • The sequential control is the most direct and competitive comparison, yet it is included only as a post hoc control. It should be treated as a primary baseline.
  • Experiments use only 300 GSM8K questions and two models, without other reasoning domains, which limits empirical generality.
Questions for authors
  • To the best of my knowledge, self-consistency is typically used to scale test-time compute beyond a single trace. What is the motivation for splitting a total budget comparable to a single reasoning trace across four streams? Is there a practical setting where this regime applies?
  • Would the conclusions change under a larger total budget (e.g., 4x the typical trace length), where early stopping in self-consistency is usually applied?
2 comments

Thank you. We agree that sequential completion should be a primary baseline and have run a new longer-budget confirmation experiment. It uses 64 fixed GSM8K questions, eight independent Qwen3-4B-Thinking-2507 samples per question, and an 8192-token cap per sample. All 512 samples terminated with EOS; their mean length is 1728 tokens, so an 8192-token question budget is 4.7 times the mean single-trace length.

With four streams at this budget, fixed-order sequential completion scores 95.3% with zero missing answers and uses 5386 mean tokens. Adding two-answer agreement stopping preserves exactly the same question-level accuracy and zero-missing rate while reducing mean use to 3333 tokens; the paired saving is 2053 tokens [1787, 2315]. At the larger eight-stream, 32768-token setting, both sequential policies score 96.9% with zero missing answers, while agreement reduces mean use from 13476 to 3458 tokens, a paired saving of 10017 [8484, 11650].

We also separated missing from incorrect outcomes. In the four-stream 4096-token round-robin-agreement condition, the 64 questions contain 31 correct, 33 missing, and 0 explicit wrong outcomes. Sequential agreement at the same budget yields 59 correct, 4 missing, and 1 explicit wrong. Across the full 512 completed traces, 95.7% are correct, 4.3% are explicit wrong, and none are missing. These results confirm that the original small-budget comparison primarily measured answer availability. We will make sequential allocation primary, report missing and wrong outcomes separately, and limit the claim to token cost rather than serving latency.

I sincerely thank the authors for their response. However, my concerns about this work remain, and I believe Reviewers nicZ and ZXEG have raised similar concerns. While I agree that two-answer agreement can save tokens to some extent, its effect on accuracy remains unclear. I will therefore maintain my original score.

Reviewer_ZXEGOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper asks whether the observation that longer completed reasoning responses tend to be wrong translates into a useful stopping rule during generation. It replays four saved sampled streams per question (300 GSM8K test questions, Qwen3-4B-Thinking and R1-Distill-Qwen-7B) under a shared output budget of 2,048 or 4,096 tokens, advancing one stream at a time in 128-token blocks. It compares uniform, random and round-robin allocation with an agreement rule that stops once two visible answers match. Agreement ties round-robin exactly while using slightly fewer tokens, random allocation does better, and a post hoc sequential schedule that finishes one stream at a time does much better. Separately, a retrospective analysis shows that incorrect responses are longer than correct ones within the same question.

Critical strengths
  1. The distinction between a length–correctness association and a decision made from prefixes is correct and worth making. Replaying the same saved streams for every policy is a clean way to compare schedulers.

  2. The paper is careful about what it measures. It separates replay cost from corpus generation cost, keeps the original analysis next to the amended one, and says which controls were added post hoc.

Critical weaknesses
  1. The token budget is too small for the comparison to mean anything.

The shared budget gives each of the four streams about 1,000 tokens, which is less than an average response, so most streams never finish. About 40% of questions end with no answer at all under uniform, round-robin and agreement (Table 4). With so few finished answers the agreement rule rarely fires, so tying round-robin says nothing about agreement stopping. The natural version, sampling full answers one at a time and stopping when two agree, is missing, even though Adaptive-Consistency (Aggarwal et al., EMNLP 2023) already do exactly this.

  1. The paper doesn't test what its title asks.

The title asks whether completed length can guide decisions on prefixes, but no length-based rule is ever evaluated, even though how long a stream has run without an answer is visible from the prefix. So the length analysis and the agreement experiment are two separate results, and the conclusion linking them doesn't follow. The length finding itself, that wrong answers are longer, is already known for reasoning models (e.g. Zeng et al., ACL 2025).

  1. The study is too small to support its conclusions.

Everything rests on 300 GSM8K questions, two small models and one set of four samples per question. GSM8K is close to saturated for these models, so it doesn't reach the setting where stopping rules matter, i.e. hard problems with long reasoning. The paper itself notes that newly sampled responses could change the differences even on the same questions. Right now this reads more like a course project written up as a paper than an ICLR contribution. I understand that people may be resource constrained, but for an empirical finding like this, current evidence is not even close to support what the author claims.

Questions for authors
  1. What fraction of streams finish within 512 and 1,024 tokens? This would show how much of the result is fixed by the budget alone.
  2. How does agreement stopping do when combined with sequential allocation, or under budgets large enough for most streams to finish?
  3. Do the conclusions hold on a harder benchmark where these models are not near ceiling (e.g. MATH-500 or AIME), and with more than four samples?
2 comments

Thank you. We ran the requested longer-budget and sequential-agreement experiment on a new independent cohort of 64 fixed GSM8K questions, with eight Qwen3-4B-Thinking-2507 samples per question and an 8192-token cap per sample. All 512 samples terminated with EOS.

The completion fractions confirm your diagnosis. Only 4.7% of streams finish within 512 tokens and 37.9% within 1024; explicit answers are visible in 4.7% and 38.1%, respectively. The original balanced small-budget regime therefore leaves most streams unfinished.

With four streams and an 8192-token question budget, sequential completion without early stopping reaches 95.3% accuracy with zero missing answers. Sequential two-answer agreement also reaches 95.3% on exactly the same questions, while reducing mean token use from 5386 to 3333; the paired reduction is 2053 tokens [1787, 2315]. Round-robin agreement reaches 87.5% and leaves seven questions without an answer, whereas sequential agreement leaves none.

We also tested more than four samples under a much larger budget. With eight streams and a 32768-token question budget, sequential allocation scores 96.9% both with and without agreement stopping, with zero missing answers. Agreement reduces mean use from 13476 to 3458 tokens, a paired saving of 10017 [8484, 11650]. We will therefore promote sequential allocation to a primary baseline and revise the conclusion: the original result was dominated by answer availability under balanced allocation, while agreement stopping becomes useful for cost reduction once paths are allowed to finish. This confirmation remains GSM8K-specific; we will not claim validation on harder domains.

Thank you for the responses. I just don't think we can get meaningful signals from gsm8k. and the experiments just don't hold up to what the paper claims. will keep my score.

Reviewer_nicZOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper distinguishes retrospective associations between completed-response length and correctness from the effects of prefix-only budget decisions. It evaluates uniform, random, round-robin, and agreement-based policies on fixed trajectories from Qwen3-4B-Thinking-2507 and DeepSeek-R1-Distill-Qwen-7B: four rollouts per model on each of 300 GSM8K test questions. Policies reveal prefixes in blocks of up to 128 tokens under shared per-question ceilings of 2,048 or 4,096 tokens. At the larger ceiling, Agreement has identical question-level correctness to Round-robin, with accuracies of 58.0% and 59.3%, while saving approximately 131 and 235 tokens per question. Random allocation performs better, and a post hoc sequential control reaches 88.3% on both models. The authors interpret these results as evidence that answer availability and allocation must be distinguished from retrospective length associations and stopping effects.

Critical strengths
  1. Controlled paired comparisons. All policies operate on the same sampled trajectories for each question, controlling variation in the response pool. Question-level bootstrap resampling respects dependence among rollouts from the same question.
  2. Transparent scope and cost accounting. The paper separates replay-token consumption from corpus-generation costs, documents analysis amendments and post hoc controls, and explicitly avoids equating token savings with measured latency improvements.
  3. Useful answer-availability diagnostics. Reporting missing-answer rates alongside accuracy helps explain why spreading a small budget across several rollouts can fail even when the saved trajectories contain usable answers.
Critical weaknesses
  1. The motivation concerns reasoning length, but the experiments shift to scheduling rollouts within a question without directly connecting the two. The paper starts from variation in response length across problems and the relationship between length and correctness. Its experiments instead fix the budget ceiling for each question and compare how that budget is distributed among four rollouts. They neither investigate how to allocate budgets according to the problem nor use observed reasoning content to decide whether a particular rollout is worth continuing. Although Section 6.3 compares correct and incorrect rollout lengths within questions, this remains a retrospective statistic and does not inform Agreement's decisions. The experiments therefore show whether different schedules allow rollouts to reach a final answer, rather than how the motivating length observations can be translated into effective decisions during generation. The paper does not adequately explain why testing answer-agreement stopping addresses the research question motivated by length–correctness associations.

  2. Agreement amounts to round-robin scheduling with answer-agreement stopping, offering limited methodological novelty. According to Table 1, both Agreement and Round-robin select the stream with the fewest exposed tokens and break ties by sample index. While all streams remain open and each step advances a full 128-token block, this is simply round-robin advancement by index, not prioritization based on reasoning quality. Agreement adds only per-stream closure upon answer detection and global stopping once two answers agree. If answer detection and stream termination coincide at the scheduler's decision point, the first distinction disappears: Agreement becomes round-robin scheduling with a two-answer agreement stopping condition. I therefore do not see a substantive methodological innovation beyond this simple combination. Positioning the paper as an empirical audit is legitimate, but does not substitute for establishing why its additional empirical insight is sufficiently significant.

  3. The experimental setting is too simple to support a broad discussion of reasoning-budget allocation. All results come from 300 GSM8K test questions, two relatively small models—Qwen3-4B-Thinking-2507 and DeepSeek-R1-Distill-Qwen-7B—and two budget ceilings. There is no evaluation on more challenging tasks, other reasoning domains, or larger models. Notably, under the 4,096-token ceiling, simply switching to sequential rollout completion yields 88.3% accuracy on both models, versus 58.0% and 59.3% for Agreement. This gap indicates that an important bottleneck in the current setup is that balanced allocation leaves many rollouts without a final answer. It does not adequately test whether a stopping signal can distinguish paths worth continuing from paths worth stopping in more complex reasoning. Moreover, Agreement changes both per-stream closure and question-level stopping relative to Round-robin without separate ablations, leaving the token savings contributed by each operation unclear.

Questions for authors

Is Agreement operationally different from round-robin scheduling with a two-answer agreement stopping condition? If a stream terminates when its final answer becomes visible, which is the expected behavior of most LLMs, so that answer detection and stream termination coincide at the scheduler’s decision point, both methods appear to advance the same streams until agreement occurs. Under this condition, what additional mechanism, if any, distinguishes Agreement from round-robin plus agreement-based early stopping? This question concerns that augmented baseline, not plain Round-robin, which does not stop on agreement.

2 comments

Thank you for pressing on the operational definition. Yes: our original Agreement policy is round-robin scheduling augmented by two operations—closing a stream once an explicit answer is visible and stopping the question once two visible answers match. It has no additional reasoning-quality prioritizer. We will state this equivalence directly and present Agreement as an audited stopping rule, not a new scheduling algorithm.

We also ran the requested ablation on a new independent cohort: 64 fixed GSM8K questions, eight Qwen3-4B-Thinking-2507 samples per question, and an 8192-token cap per sample. All 512 samples terminated with EOS. With four streams and an 8192-token question budget, plain round-robin scores 87.5% using 5399 mean tokens. Closing streams on answer detection alone leaves accuracy at 87.5% and saves only 4 tokens on average. Adding global two-answer stopping still scores 87.5% and uses 4976 tokens; relative to closure alone, the paired saving is 419 tokens [290, 570]. Thus nearly all savings come from question-level agreement stopping, not per-stream closure.

The natural sequential baseline is stronger. Completing streams in fixed order scores 95.3% with no missing answers. Adding the same agreement stopping preserves exactly the same question-level accuracy while reducing mean use from 5386 to 3333 tokens, a paired saving of 2053 [1787, 2315]. We will make sequential-plus-agreement a primary comparison and narrow the contribution to diagnosing how allocation and stopping interact.

Thanks for the response. However, the feedback doesn't address my concerns and I will keep my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.