acceptodds
Under review as a conference paper at ICLR 2027

What Do Early Answers Reveal? Readout Protocols and Post-Answer Revision

Abstract

Does a correct continuation from an early reasoning prefix mean the model was ready to stop? We compare free continuation (no extra request) with continuation prompted to give a final answer, scoring both first and last answers. Across Qwen3 and GPT-OSS on MATH-500 and GPQA-Diamond, recovery depends on the request and which answer is scored. On 100 MATH problems, Qwen3-8B with thinking enabled at a 10% prefix achieves 69.3% free last-answer accuracy; requested outputs rise from 31.8% at the first answer to 73.8% at the last. Among improvements, the median delay from the first answer to the first later correct box is 1,258 tokens. Allowing 16,384 reference tokens on 50 reused questions preserves a 52-point first-to-last gain, but no clear last-answer advantage for either procedure. Closing the reasoning channel reduces later gains in several settings, but not clearly for Qwen at 50%. At 50%, equal continuation budgets retain a Qwen prefix benefit with thinking, but not a clear benefit without it. An online case study saves tokens in the main generation while increasing time and total generation. Recovery, prefix benefit, and stopping efficiency therefore require separate evidence: score first and last answers, compare question-only recovery, and count the full cost before interpreting recovery as readiness to stop.

Primary area:
interpretability and explainable AI
Keywords:
chain-of-thought reasoning, black-box evaluation, answer extraction, self-consistency, early-exit inference

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_VSjCOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper compares forced answering to free continuations in LLM reasoning, in math and science settings. They show that a forced answer at the beginning of reasoning differs from the eventual answer later on, and is not always predictive of the final answer. They run experiments from partial chain-of-thought, showing that a larger test-time budget for continuations improves performance. The paper also includes a resampling experiment to measure the relation between the forced answer at various points of the rollout with the distribution of final answers from free continuations to see if one could have ended the reasoning trace there.

Critical strengths

None

Critical weaknesses

The paper's premise, that a forced answer due to an insertion in chain-of-thought does not correspond with readiness to stop, is self-evident. A forced answer occurs in the first place because of the insertion, not due to the model ending the reasoning trace. Measuring forced answer accuracy against free continuation accuracy (what reasoning LLMs are trained to do from RL) is not a comparison worth making in the first place. Prior work using forced answer accuracy measures whether a thinking budget is necessary for a particular problem, and the results of this paper add no new claims or nuance on top of this discussion.

The main finding in section 4.1 shows that an initially incorrect answer can be corrected with more chain-of-thought, which has been well-documented in the literature of test-time scaling. Additionally, in section 4.4 (Table 2), the authors match only additional generation tokens, not total computation, so the observed advantage of starting from a reasoning prefix is consistent with ordinary test-time scaling rather than demonstrating a novel benefit of the prefix. The prefix is anyways on-policy, so it is unclear exactly what claim the authors would be making in the first place, as it would be the same as continuing the original chain-of-thought.

Questions for authors

What is the purpose of comparing forced answer to readiness to stop? The intervention encourages the model to stop reasoning in the first place, as there is an insertion indicating a final answer.

Why use the a full continuation resampling as a method to claim that the forced answer was correct or incorrect in the case study? How does this function as a deployment case study if there are several parallel generations to verify a forced answer? How is this different than the forced answering experiments done in prior work (Lanham et al. 2023, Boppana et al. 2026)?

Why use the a full continuation resampling as a method in the deployment case study in §4.6? The proposed stopping policy requires eight parallel continuations at each checkpoint, making its increased inference cost unsurprising. How does this experiment meaningfully evaluate the efficiency of early stopping, rather than performing even more inference through free continuations? How does the paper's evaluation of forced-answer readouts meaningfully extend the experiments in Lanham et al. (2023) and Boppana et al. (2026)?

Reviewer_wYd9OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper examines what successful answer recovery from an early reasoning prefix actually establishes. It compares free and answer-requested continuations, scores their first and last answers, and studies sensitivity to generation budgets, channel closure, and answer extraction. Experiments on Qwen3 and GPT-OSS show substantial post-answer revision in several settings. Question-only controls and an online stopping case study further distinguish prefix benefit from practical efficiency.

Critical strengths

The strongest aspect is the careful accounting of what happens after the first answer. The transition analysis and correction-token measurements make clear that final-answer accuracy can conceal substantial additional generation. The authors also report exceptions and sensitivity analyses rather than presenting the protocol-dependent ranking as a universal result. These checks lend credibility to the paper’s narrow empirical conclusions.

Critical weaknesses

The contribution beyond closely related work is limited. Budget-confounded prefix recovery and the limitations of agreement-based stopping are already addressed by Bulut (2026) and Mo et al. (2026), respectively. The main additional contribution here is a detailed first-versus-last-answer audit. The measured effects are substantial, but the conceptual implication is fairly direct: requesting an answer while allowing continued generation does not measure immediate answering ability. The paper documents this distinction thoroughly, yet offers limited additional understanding of when early answers are informative or how to construct a better readout. My concern is the significance of the contribution, rather than the correctness of the reported effect.

The practical consequences for existing methods remain unestablished. The study does not directly reassess representative published methods under their actual prompting, stopping, and scoring protocols. Its main forced-answer setting permits long continuations, whereas methods that terminate after a short trial answer may measure a different quantity. Likewise, the online experiment evaluates an untuned rule that waits for eight potentially long free continuations; its poor efficiency is informative about that implementation but says little about competitive stopping policies. The authors acknowledge these boundaries, but they leave the paper without a convincing demonstration that its audit materially changes an existing scientific conclusion or practical method.

Questions for authors

Which specific published result would change under the proposed first/last-answer audit while preserving that method’s original protocol? A concrete demonstration would materially affect my assessment of the contribution.

Reviewer_KD46OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether obtaining a correct answer from an early reasoning prefix provides evidence that a model is ready to stop. The authors compare free continuation with answer-requested continuation, evaluating both the first and last answers generated under each protocol. Experiments on MATH and GPQA-Diamond with Qwen3-8B and GPT-OSS-120B demonstrate that substantial answer revision can occur after an initial answer has been produced. Additional controls examine the effects of continuation budgets, reasoning-channel closure, reference truncation, and question-only baselines. An online stopping experiment further shows that reducing tokens in the main generation does not necessarily reduce total computation or latency when probing costs are included.

Critical strengths
  1. Careful analysis of what early-answer evaluations actually measure.

The distinction between first-answer correctness, post-answer revision, and final-answer correctness is important. The experiments demonstrate that a correct final answer following an early prefix may depend on substantial additional generation. The analysis helps prevent early-answer recoverability from being incorrectly interpreted as evidence that the model was already ready to stop.

  1. Useful controls for evaluation protocols and computational costs.

The experiments systematically examine how answer extraction, continuation budgets, reasoning-channel transitions, and reference termination affect the conclusions. The question-only comparisons and online stopping experiment are particularly valuable because they distinguish answer recovery from prefix-specific benefits and practical stopping efficiency.

Critical weaknesses
  1. The incremental contribution relative to recent work is somewhat limited.

Recent studies have already questioned whether early-answer agreement or successful prefix continuation provides reliable evidence for stopping, including the effects of additional computation and problem difficulty. The present paper contributes a more detailed first-to-last answer audit, but the main practical conclusion (=early-answer recovery does not necessarily establish stopping readiness) substantially overlaps with these existing findings. It is not sufficiently clear how the new analysis changes the understanding of early stopping beyond providing a more detailed characterization of the same limitation.

  1. The experimental design provides limited evidence about practical stopping decisions.

The main offline analysis defines checkpoints as fractions of observed reference lengths, which are not available at inference time. A substantial portion of the Qwen thinking references also reaches the reference-token limit. Although the authors include longer-reference controls and an online experiment, the online evaluation considers only one relatively expensive agreement-based stopping rule. Consequently, the negative latency result is informative for that implementation but does not establish the effectiveness or inefficiency of more competitive stopping strategies.

Questions for authors
  • Novelty relative to prior work. What substantive conclusion about early stopping becomes identifiable through the proposed first-to-last answer analysis that cannot be established by existing matched-budget prefix evaluations or self-consensus studies?

  • Practical applicability: How well do the main findings transfer to fixed absolute-token checkpoints, rather than checkpoints defined retrospectively as fractions of reference length?

  • Stopping efficiency. The online experiment evaluates an expensive eight-probe agreement rule. Can the authors clarify what conclusions can be drawn about stopping efficiency beyond this specific rule, and whether substantially cheaper probing would change the practical interpretation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.