acceptodds
Under review as a conference paper at ICLR 2027

Answer Interfaces Change the Measured Cost of Early Stopping

Abstract

Early-exit methods are commonly compared by reasoning tokens saved and accuracy lost, as if the loss belonged to the stopping rule. Yet a stop returns a reasoning prefix; a terminal interface must still turn it into a scored answer. We hold rule-selected prefixes fixed and cross the interfaces that read them. On Qwen3.5-9B, the same Answer Convergence (AC) stops lose accuracy points under a staged cascade inside the open thinking block and after model-specific closure. Stopping changes how often the answer stage sees an open, unresolved prefix, while requests, prefills and closed generation respond differently to that state. At matched mean reasoning length and a 1K answer allowance, AC trails a fixed-token cut by points under the cascade but leads by after closure. In matched-prefix controls, two closed readouts agree within a pre-specified margin at 2K. The dependence recurs across caps, tasks and checkpoints, while the response to closure and prefill differs across model families. We therefore report complete systems under their own procedures and compare rules conditionally on a specified answer interface.

Primary area:
datasets and benchmarks
Keywords:
early stopping, reasoning models, answer interfaces, evaluation methodology, test-time compute

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_uHoAOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies how different methods for producing final answers from reasoning prefixes affect comparisons between early-stopping techniques. As I understand it, no method consistently performs best, and the optimal choice depends on the budget allocated to generating the final answer.

Critical strengths

None

Critical weaknesses

The paper is very difficult to follow. Despite being very familiar with the area, I couldn't understand the paper's contribution from the abstract.

The experimental design is also hard to follow, even though the experiments are simple in nature.

While I believe that careful and accurate comparisons of early-stopping techniques and their combinations with methods for producing final answers are important, I think that the current revision of this paper fails to communicate any important findings.

The presentation makes it difficult to distinguish the main findings from other conclusions.

Questions for authors

I don't have questions whose answers could materially affect my assessment.

1 comment
AuthorsOpenReview ↗

Clarifying the main finding and experimental design

We have revised the paper around one question: why can the same stopping decisions yield different measured accuracy losses? The comparison below uses the same 200 Qwen3.5-9B/AMC23 runs, stopping decisions, scorer and 1,024-token answer allowance. Only the answer procedure changes. AC here is the blank-line adaptation of Answer Convergence.

Answer procedureFull accuracyAC policy accuracyFull−AC loss
Staged request inside the thinking block80.5%45.0%35.5 pp
Free answering after native closure72.5%65.5%7.0 pp

The reported stopping loss falls by 28.5 percentage points (simultaneous 95% CI [12.5, 44.5]). Of this reduction, 20.5 points come from better answers under the AC policy, while 8.0 come from a lower Full score. Both procedures keep naturally completed answers, so Full changes only on runs that reach the reasoning limit and still require an answer. A smaller stopping loss can therefore partly reflect a weaker reference answer, even though the stopping decisions are unchanged.

Pairing each cut with its own full trajectory also lets us isolate the change at the cut. Among the 92 interrupted runs that would have ended naturally, Full is identical under the two procedures, and native closure gives a net 29 additional correct answers at the cuts (Sections 2.1–2.2).

We also tested two closed 2K procedures that meet the 95% validity threshold at Full, under each policy and among triggered stops. Both retain a small estimated AC lead over the length-matched fixed cut, while the estimated Full−AC loss changes from 5.5 to 8.0 pp (exploratory pointwise 95% CI for the paired change [0.0, 5.5], Table 2 and Appendix C.2). This separates a change in the measured stopping loss from a change in which rule ranks higher. We recommend reporting both endpoint accuracies on the same runs, stating whether naturally completed answers were kept or regenerated, and counting probes and answer generation when comparing computational cost.

We have reorganized the paper around this argument. The revised title, What Does the Accuracy Loss from Early Stopping Measure?, and abstract state this question. Figure 1 displays the four scores on the first page, and Section 2.1 works through them before introducing the equation. Section 3 examines when a stopped trace can still yield a correct answer, Section 4 compares rule rankings, and Section 5 accounts for system cost.

Reviewer_EzD4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines how the procedure used to produce a final answer affects the measured accuracy cost of early stopping. The authors keep the retained reasoning prefixes fixed and compare different answering procedures, including continuing inside the thinking block, closing the block, and prefilling the answer.

On AMC23, the same Answer Convergence stops lose 35.5 accuracy points under the staged cascade but only 7.0 after closing the thinking block. Changing the interface also reverses the comparison with a fixed cut matched for mean reasoning length. Additional experiments examine answer budgets, stopping locations, model families, and scoring procedures. The main takeaway is that early-stopping results depend on the complete answering procedure, so comparisons between stopping rules must specify these conditions.

Critical strengths

A controlled comparison that supports the main claim. The interfaces receive the same retained reasoning, and their effects are compared at both early stops and full endpoints. This helps establish that the answering procedure changes the accuracy loss attributed to stopping. Paired problem-level bootstraps and simultaneous intervals strengthen the evidence, while the larger MATH-500 experiment shows that the effect also occurs when most full traces finish naturally.

Useful experiments explaining what drives the effect. The authors separately examine the end-of-thinking token, the budget sentence, answer prefilling, and the available answer budget. Random-cut and empty-prefix controls help clarify the contribution of the retained reasoning. The different behavior across model families also gives the explanation appropriate limits. These analyses provide practical guidance about which parts of an answering procedure need attention.

Critical weaknesses

1.The practical scope of the headline reversal needs clarification. The clearest reversal uses a 1K answer allowance and a cascade inside the open thinking block, where valid-answer failures are frequent. Most audited published methods already close the block or prefill an answer. Moreover, the two tested closed readouts agree within the specified margin at 2K on the pooled control settings. It remains unclear which ranking changes persist between interfaces that pass the paper’s validity screen, with their total costs taken into account.

2.The main comparisons use materially adapted stopping rules. The EAT implementation follows an older version, and its matched threshold makes almost every exit occur at the end of the warm-up period. Answer Convergence checks blank lines instead of sentences, which substantially changes when it stops. The follow-up experiments are helpful, but the resulting comparisons still provide limited evidence about the exact published procedures.

3.The MATH-500 shared-readout validation is unfinished. Appendix B.4 reports that the entry parser accepted the placeholder “...” as an answer and incorrectly skipped some requests. The authors clearly disclose this issue and label the recorded comparison as diagnostic. However, the affected requests have not been rerun, and no recorded common readout passes the validity screen. Completing this correction would make that part of the evaluation more useful.

Questions for authors

1.Does the AC-versus-fixed-cut reversal in Table 1 persist at a 2K answer allowance using interfaces that pass the validity screen? Please also report the answer-generation and probing costs.

2.On at least one primary setting, do the main conclusions hold using the revised EAT rule and the exact released AC procedure?

3.After correcting placeholder acceptance and running the skipped MATH-500 requests, does a common readout pass the validity screen, and what is the corrected AC–EAT accuracy difference?

1 comment
AuthorsOpenReview ↗

Additional comparisons, cost accounting, and the MATH-500 correction

Q1. The AC-versus-fixed reversal did not recur between the two closed 2K procedures at the original AMC23 stops. Table 2 reports the same 40 problems and five seeds:

2K procedureFullBlank-line ACFixedAC−fixed, pp [95% CI]Lowest validity
Closed request92.5%87.0%84.5%+2.5 [−2.5, +7.5]190/200, fixed policy
Closed prefill91.5%83.5%82.5%+1.0 [−3.0, +4.5]195/200, Full

Gap intervals are pointwise; the paired change is −1.5 pp (simultaneous 95% CI [−6.5, +3.5]). Minimum validity covers Full, both policies and each rule's triggered stops. Both 2K procedures generate a fresh answer at every endpoint; the original 1K procedures retained naturally completed answers. The prefill supplies the boxed opening used by blank-line AC's trial (Appendix Table 15).

Costs include stop-selection probes and final answering:

Mean per runRequest, ACRequest, fixedPrefill, ACPrefill, fixed
Final-answer output (tokens)492.6537.53.73.8
All generated output (K tokens)6.536.566.046.03
Token work with KV reuse (K)6.876.766.376.21
Token work with cold rereading (K)59.8812.9559.3812.40

AC makes 16.66 probes per run on average, generating 65.3 tokens and processing 47,012 input tokens before reuse; the fixed cut makes none. With input and output tokens weighted equally, AC uses 1.02× the fixed cut's estimated work under the closed request with KV reuse, and 4.62× with cold rereading. K denotes 1,000 tokens (Figure 5; Appendix Table 21).

Q2. With released stopping components, the answer procedure can still change measured stopping loss; the Qwen comparisons with high submission validity retain the same ranking. Qwen3.5-9B uses AC's released sentence checks, trial reader and ten-trial rule with a Qwen-adapted trial prompt. On DeepSeek-R1-Distill-Llama-8B (R1), released AC code generates trajectories, trials and stops for all 500 MATH-500 problems; two logits processors were adapted to vLLM V1. EAT uses its released entropy probe and bias-corrected v3 controller without warm-up. Table 3A compares these stopping components with fresh answers and a common task scorer. Released AC instead returns the selected trial and uses an LLM judge; Table 3B includes its trials under our scorer (Appendix Table 16).

ComparisonAC−EAT, requestAC−EAT, prefillChange, pp [simultaneous 95% CI]Minimum validity, request / prefill
Qwen, original grid, primary 2K−27.5 pp−23.0 pp+4.5 [−3.0, +12.0]95.5% / 97.5%
Qwen, near length, secondary 4K+11.0 pp+11.5 pp+0.5 [−9.5, +10.5]95.0% / 97.5%
R1, near length, primary 2K+8.2 pp−1.2 pp−9.4 [−15.4, −3.4]66.2% / 99.2%

On Qwen, changing EAT's operating point has the larger effect. AC retains 470 reasoning tokens on average, versus 3,387 for the closest original EAT grid point and 493 for the near-length threshold. We selected the closer threshold using retained length alone, before final answering. At near-matched length, both 4K procedures pass the 95% validity screen and give AC similar leads, but AC uses 9.7× EAT's estimated token work under the request with KV reuse. At 2K, request validity at triggered EAT stops is 189/199, just below 95%.

On Qwen, sentence AC gives 63 correct answers under request and 60 under prefill, against Full counts of 185 and 183 out of 200: losses of 61.0 and 61.5 pp (Figure 4). High submission validity thus coexists with a large stopping loss.

On R1 (about 841 reasoning tokens per rule), the request has low EAT submission validity, so the gap change includes answer-production failures. Under prefill, minimum validity is 99.2%, and the −1.2 pp gap has a pointwise 95% CI of [−5.4, +3.0]. Table 3B also shows how re-answering Full changes the stopping-loss estimate: Full−AC is +20.6 pp with original Full answers and selected AC trials, −6.0 pp with a fresh 2K request, and +21.2 pp with prefill. Of 500 Full requests, 130 terminate immediately, including 109 whose original answer was correct (Appendix C.3).

Q3. Yes. We completed all 68 skipped requests. The repaired 2K strict-closed procedure passes the 95% screen on the original Qwen MATH-500 level-5 subset. Across 635 runs (127 problems, five seeds), the lowest validity over Full, all policies and triggered stops is 543/570 (95.26%), at triggered blank-line AC stops. The corrected AC−EAT-v1 differences are −0.47 pp at the matched threshold and −2.05 pp at the designated threshold (Appendix C.4).

The parser had accepted the literal “...” from a quoted formatting example and therefore skipped the answer request. Table 19 reports the added outcomes and final validity counts. Table 7 separately identifies the original 1K cascade result affected by the same parser error.

Reviewer_QhqXOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates how the method used to convert a truncated reasoning trace into a final answer changes the measured accuracy–efficiency trade-off. Demonstrates that conclusions about early-stopping algorithms can reverse depending on this terminal interface.

Critical strengths

The central conceptual distinction between stopping policy and answer interface is well founded. Holding prefixes fixed while varying the interface provides a clean identification, supported by controls, replications, and explicit statistical testing.

Critical weaknesses

The largest effects arise from requesting answers inside an open reasoning block, a potentially unrealistic or flawed evaluation procedure. Most surveyed published methods already close or prefill the answer, and the strongest ranking reversal is specific to certain models, tasks, and budgets.

Questions for authors

Do the major conclusions still hold when restricting attention to realistic, well-implemented terminal interfaces? Does this reveal a general problem with early-stopping evaluation, or principally demonstrate the consequences of an inappropriate answer-generation protocol?

1 comment
AuthorsOpenReview ↗

Closed answer procedures and practical scope

Thank you for raising the practical scope of the result. We compared two closed answer procedures at the original AMC23 stopping points.

The original AMC23 ranking reversal did not recur between the two closed 2K procedures. Blank-line Answer Convergence (AC) leads the length-matched fixed cut by an estimated 2.5 percentage points (pp) under a closed request and 1.0 pp under a closed prefill. Both procedures yield at least 95% valid submissions at Full, across each policy, and at each rule's triggered stops. Both individual gap intervals include zero; the paired change in the gap is −1.5 pp (simultaneous 95% CI [−6.5, +3.5], Table 2). The larger 1K reversal occurs between procedures with frequent submission failures, which the revised comparison now makes explicit.

The stopping loss also depends on what the answer procedure does to the full-run reference. In the same high-validity 2K comparison, switching from request to prefill changes AC's correct count from 174 to 167 and Full's from 185 to 183. Full−AC therefore rises from 5.5 to 8.0 pp (paired change +2.5 pp, exploratory pointwise 95% CI [0.0, 5.5], Appendix C.2). Both procedures generate a fresh answer even from naturally completed runs. Reporting the two accuracies makes the source of this change explicit; the interval leaves uncertainty about its size.

In a separate budget control on fixed, open AC inputs without a recognized answer, strict closed and native closure agree within the prespecified ±5 pp margin at 2K and 4K when the three settings are equally weighted (Figure 3). On MATH level 4, two prefills with minimum validity near 99% yield a smaller, exploratory change of −3.41 pp in AC−fixed (pointwise 95% CI [−6.02, −1.14]). The cues differ in wording and block state. This pair was selected from 21 pairs in each of five settings; the interval is unadjusted, and both individual rule-gap intervals include zero (Appendix C.4).

We therefore recommend reporting Full and stopped-policy accuracy separately, stating whether naturally completed answers were kept or regenerated, and reporting submission validity alongside each rule comparison.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.