acceptodds
Under review as a conference paper at ICLR 2027

Shorter Reasoning, Earlier Answers? An Evaluation of Reasoning Interfaces

Abstract

Large language models often reason at length before answering, increasing cost and latency. We evaluate two reasoning interfaces in different model families under a shared design. Qwen3 receives concision and budget instructions, and gpt-oss uses its trained reasoning effort settings. These interfaces can shorten traces, but completed responses alone do not establish whether an interface improves earlier answers or changes when the model finishes. We evaluate paired runs at shared reasoning allowances and distinguish the answers models generate from the answers we read off their unfinished reasoning. The main comparisons cover Qwen3-14B and gpt-oss-20b/-120b on 198 GPQA Diamond and 500 MMLU-Pro questions. Across seven shared allowances, low gpt-oss effort improves generated-answer accuracy over high effort by 4.5-18.9 percentage points at 512 tokens, while high effort is 4.4-8.8 points more accurate at 8,192 tokens. The early advantage is concentrated in pairs where lower effort has completed and high effort remains active. For Qwen3-14B on MMLU-Pro, a concise instruction improves generated-answer accuracy by 3.5 points at 512 tokens and 2.5 points at 2,048. On pairs that both policies leave unfinished at 512 tokens, it still improves generated answers by 4.1 points. A budget instruction that announces the allowance shortens reasoning but gives smaller and less consistent gains, and matching the announced allowance to the evaluation horizon adds little in the tested comparisons. Neither the effort advantage nor the concise gain reflects adaptation to a stated allowance, and neither establishes uniformly better unfinished reasoning.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, reasoning efficiency, test-time compute, token budgets, chain-of-thought, evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_S8nJOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper empirically studies reasoning-control interfaces in Qwen3 and gpt-oss under fixed inference budgets. It compares concise/budget prompting and reasoning-effort settings, distinguishing naturally completed answers from answers obtained by interrupting unfinished reasoning. The results show that lower-effort or concise settings can perform better under short budgets, while higher effort becomes stronger with larger budgets. The analysis suggests that part of the early advantage comes from earlier completion rather than uniformly better intermediate reasoning, and that explicitly stating a token budget provides little additional benefit.

Critical strengths
  1. The paper clearly distinguishes reasoning-length control from budget-aware reasoning adaptation, showing that models may learn to reason less without actually learning how to allocate reasoning under a specified budget.

  2. The paper exposes a meaningful completion-time confound in fixed-budget evaluation: short-budget gains can partly reflect earlier answer completion rather than uniformly better intermediate reasoning.

Critical weaknesses
  1. The paper is primarily a careful empirical characterization rather than a substantive methodological or mechanistic contribution. Although it reports several useful observations about reasoning control, completion timing, and budget-aware behavior, these findings remain largely descriptive. The paper does not provide a deeper explanation of the reasoning mechanisms that lead to these effects, nor does it introduce a new method, theoretical framework, or formal model.
Questions for authors

None

1 comment

Thank you for asking for a clearer account of what the evaluation teaches us. We have reorganized the paper around the clause experiment and the finding it supports.

What the clause comparison shows

The new Qwen3-14B study tests concision and answer priority separately and together on 1,000 MMLU-Pro, 198 GPQA Diamond and 500 open-ended mathematics items. At 512 reasoning tokens, adding concision improves accuracy by 2.46 percentage points on MMLU-Pro (simultaneous 95% band [1.17, 3.76]) and 2.90 on mathematics [0.45, 5.35], averaging over whether answer priority is present.

The average concision effect is positive at two MMLU-Pro allowances and all three mathematics allowances under the complete 45-contrast family. Direct comparisons between the single-clause prompts favor concision at MMLU-Pro 512 and mathematics 2,048 and 8,192. The average answer-priority effects and interactions remain uncertain.

Why completion and the answer procedure matter

On mathematics at 512 tokens, answer priority increases reasoning completion by 3.98 points [2.95, 5.03], while concision changes it by −0.83 [−2.40, 0.78]. These are average clause effects with pointwise 95% intervals. Completion and accuracy therefore give different information about the instruction. This is evidence about observable behavior, not an internal mechanism or a causal effect of stopping alone.

The primary accuracy procedure regenerates final answers from reasoning prefixes, including naturally completed reasoning, with 64 answer tokens for multiple-choice questions and 256 for mathematics. Changing the final-answer context or allowance changes the measured prompt gain. The new mathematics grades are AI-reviewed, with corrected references and bounds for identified ambiguity, without human validation.

Contribution and clearer presentation

The controlled 14B experiment identifies accuracy gains from the concision clause. Separate checks of the original two-clause prompt show how its benefit varies across models, tasks and answer procedures. The identical combined prompt produces both gains and losses across Qwen sizes. Lower natural token use also accompanies accuracy losses on MMLU-Pro and GPQA. With independently measured reference difficulty, we find larger gains from the combined prompt on easier MMLU-Pro questions for Qwen3-14B, while the mathematics difficulty comparison remains unresolved. The earlier announced-budget matching tests remain separate and leave room for small matching benefits.

We build on Sun et al.’s strict-budget evaluation and Ballon et al.’s trajectory probing. Qwen3 prompt instructions and gpt-oss effort settings provide two interface case studies, with controlled policy comparisons within each model and different families and answer procedures across interfaces.

The revised main text introduces the question and primary answer procedure, then presents the clause comparison, model/task scope and natural accuracy–cost trade-offs. Earlier matching, completion-state and answer-probability analyses remain in the historical appendix. This gives the reader the finding and the evidence needed to interpret it before the detailed diagnostics.

Reviewer_RfUkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how different reasoning control mechanisms affect answer accuracy and completion behavior under limited reasoning-token budgets. The authors evaluate prompt-based reasoning control in Qwen3 and reasoning effort settings in gpt-oss, using a shared evaluation framework that forces models to generate answers at predefined reasoning-token budgets.

The main experiments focus on Qwen3-14B, gpt-oss-20b, and gpt-oss-120b, evaluated primarily on GPQA Diamond and MMLU-Pro. Additional experiments involving smaller Qwen3 models and Omni-MATH-2 are presented in the appendix.

The results show that lower reasoning effort in gpt-oss achieves higher accuracy under short reasoning budgets, whereas higher effort becomes more advantageous as the available budget increases. For Qwen3, concise reasoning instructions improve early-answer accuracy, while explicitly specifying a numerical token budget provides limited additional benefits.

The paper makes a meaningful contribution by distinguishing shorter reasoning traces, earlier answer generation, and improvements in unfinished reasoning quality. However, I have concerns about the generalizability of the findings across different models and benchmarks, as well as the lack of a systematic analysis of how problem difficulty affects the relationship between reasoning budget and answer accuracy.

Critical strengths

S1. Clear motivation and a meaningful evaluation perspective

The paper addresses an important limitation of conventional reasoning-efficiency evaluations, which typically focus on final-answer accuracy and total generated tokens.

By evaluating models under shared reasoning-token budgets and explicitly considering completion behavior, the authors provide a useful perspective on reasoning efficiency in resource-constrained inference settings.

S2. Analysis of reasoning completion states

The authors distinguish cases where both policies remain active from cases where one policy has already completed its reasoning.

This analysis provides useful insights into why lower reasoning effort can achieve higher accuracy under short token budgets. In particular, the findings suggest that earlier completion contributes substantially to the observed accuracy advantage, rather than necessarily indicating superior intermediate reasoning quality.

S3. Statistical analysis and additional replication experiments

The paper employs paired comparisons, bootstrap confidence intervals, and analyses across multiple reasoning budgets. These efforts strengthen the reliability of the reported empirical observations.

In addition, the authors replicate the main Qwen3-14B prompt-based results on a separate set of 500 MMLU-Pro questions that were not used during prompt selection. This is a valuable step toward validating the experimental findings.

Critical weaknesses

W1. Limited generalizability across model architectures and scales

The paper provides insufficient evidence that its main findings generalize consistently across different model architectures and sizes.

Although the experiments include Qwen3-4B/8B/14B and gpt-oss-20b/120b, the main prompt-based reasoning control experiments primarily focus on Qwen3-14B. Similarly, the reasoning effort experiments are restricted to two models from the same gpt-oss family.

Appendix Table A31 presents additional results for Qwen3-8B, but these experiments do not demonstrate consistent accuracy improvements under the Numeric/Concision instruction. However, the evaluated instruction differs from the concise prompt that produces the strongest results for Qwen3-14B. Therefore, these experiments do not directly establish whether the main concise-prompt effect generalizes across model sizes.

Furthermore, Qwen3 and gpt-oss are evaluated using different reasoning control mechanisms. Consequently, it is difficult to separate the effects of model architecture, training, and reasoning control interfaces.

While the reported results provide useful empirical observations for the evaluated settings, the evidence remains insufficient to support broader conclusions about reasoning behavior across LLMs.

I suggest evaluating the same concise reasoning instruction across multiple Qwen3 model sizes and, if possible, examining whether similar effects can be reproduced in additional model families under comparable reasoning control settings.

W2. Lack of systematic analysis of problem difficulty and its interaction with reasoning budgets

A major limitation is the absence of a systematic analysis of how problem difficulty affects the effectiveness of reasoning control.

The paper demonstrates that lower reasoning effort or concise prompting can be advantageous under short reasoning budgets, while these advantages diminish or reverse as the reasoning budget increases.

However, this relationship may depend strongly on problem difficulty.

For relatively easy questions, concise reasoning may be sufficient to reach the correct answer. In contrast, complex problems requiring multiple reasoning steps may benefit substantially from longer reasoning trajectories, and prematurely terminating the reasoning process could reduce accuracy.

Although the authors use difficulty-stratified sampling for Omni-MATH-2, they do not report performance separately across difficulty levels. Similarly, Appendix Table A32 presents category-level accuracy differences for MMLU-Pro, but subject categories should not be treated as equivalent to problem difficulty.

This issue is particularly relevant to the completion-state analysis. If easier questions tend to be completed earlier, conditioning on both policies remaining unfinished may introduce selection bias, potentially changing the difficulty distribution of the evaluated subset.

The authors acknowledge this selection issue, but do not explicitly control for problem difficulty.

I believe the following analyses would substantially strengthen the paper:

Divide benchmark questions into difficulty groups (e.g., Easy, Medium, and Hard) using an independent difficulty measure.

Compare accuracy and completion rates across reasoning budgets within each difficulty group.

Examine whether the relative advantages of Low/High effort or Concise/Ordinary prompting change significantly with problem difficulty.

Investigate whether the reasoning budget at which high effort overtakes low effort differs across difficulty levels.

Such analyses would provide a more informative explanation of when and why shorter reasoning is beneficial, rather than only demonstrating that shorter reasoning can improve early-answer accuracy in certain evaluation settings.

W3. Inconsistent effects across benchmarks and limited cross-task validation

The effectiveness of reasoning control does not appear to be equally consistent across the evaluated benchmarks.

For example, Appendix Table A32(b) reports that the concise instruction improves Qwen3-14B accuracy by approximately 4.9 percentage points on MMLU-Pro at a 512-token budget, whereas the corresponding improvement on GPQA Diamond is only approximately 0.2 percentage points.

Although these results do not establish a statistically significant difference between the two benchmarks, they suggest that the effectiveness of concise reasoning may depend on task characteristics.

The Omni-MATH-2 experiments are a useful extension beyond multiple-choice questions. However, the Qwen3 experiments on this benchmark evaluate the Numeric/Concision instruction rather than the concise instruction that yields the strongest main results. Moreover, these experiments use only a single rollout per condition.

Consequently, the additional benchmark results provide limited evidence that the main Qwen3 findings generalize to different reasoning tasks.

Given the paper's broader motivation of evaluating reasoning efficiency, a more systematic analysis across benchmarks would improve the strength of its conclusions.

In particular, it would be useful to investigate whether differences across benchmarks can be explained by problem difficulty, task structure, or the amount of reasoning required to obtain a correct answer.

W4. Interpretation of early accuracy improvements

The interpretation of early-answer accuracy improvements also deserves further clarification.

The authors appropriately distinguish earlier completion from improvements in unfinished reasoning quality and acknowledge that the completion-state decomposition does not establish a causal stopping-only mechanism.

Nevertheless, some of the observed improvements in the Qwen experiments are partly attributable to differences in answer parseability.

This raises an important question regarding the extent to which early-answer improvements reflect better information acquisition during reasoning, changes in reasoning termination behavior, or improvements in final-answer generation and formatting.

A clearer discussion or additional analysis separating these factors would strengthen the interpretation of the experimental results.

Questions for authors

Q1. Generalization across model sizes

Does the early-answer accuracy improvement observed with concise prompting in Qwen3-14B also hold for Qwen3-4B and Qwen3-8B?

The additional experiments in Appendix Table A31 use a different prompt condition, making it difficult to determine whether the main concise-prompt effect generalizes across model scales.

Q2. Interaction between problem difficulty and reasoning budget

Have the authors investigated whether the relative performance of low and high reasoning effort depends on problem difficulty?

Since Omni-MATH-2 provides difficulty-related metadata, it may be possible to conduct a difficulty-stratified analysis using the existing experimental results.

In particular, I would be interested in whether the reasoning budget at which high effort overtakes low effort differs between easy and difficult questions.

Q3. Differences across benchmarks

Appendix Table A32(b) indicates that concise prompting provides a substantially larger improvement on MMLU-Pro than on GPQA Diamond at a 512-token reasoning budget.

Do the authors have an explanation for this difference?

Could it be related to differences in problem difficulty, task characteristics, or other benchmark-specific factors? It would be helpful to provide an additional statistical analysis or discussion of these observations.

Q4. Distinguishing earlier completion from improved reasoning quality

A substantial portion of the early accuracy advantage appears to be concentrated in cases where one policy has already completed its reasoning.

How can the authors further distinguish improvements due to earlier completion from improvements in the quality of reasoning under the same token budget?

In particular, would the observed differences remain consistent after controlling for problem difficulty?

2 comments

Thank you for distinguishing a same-prompt test from an extension with a different instruction. We have completed the requested same-prompt, independent-difficulty, and final-answer comparisons.

Same prompt across model sizes (W1/Q1)

Qwen3-4B, -8B and -14B received identical ordinary and combined instructions on 1,000 MMLU-Pro and 198 GPQA Diamond items, with eight generations per item and condition. Reasoning allowances and answer procedures were identical. The 30-contrast simultaneous family includes direct between-model effect differences.

At 8,192 tokens on MMLU-Pro, accuracy changes were 0.55, 11.55 and 0.80 percentage points for 4B, 8B and 14B. The 8B-minus-14B effect difference was 10.75 [7.91, 13.59] points. On GPQA, effects were −6.76 for 4B at 512 [−12.09, −1.42] and +13.45 for 8B at 8,192 [6.05, 20.85]. These are simultaneous 95% bands. The same prompt yields both gains and losses, without a monotonic model-size pattern. The independent 500-question replication remains separate prior evidence.

Independent difficulty (W2/Q2)

We estimated multiple-choice difficulty from 16 independent ordinary Qwen3-14B natural answers per item, using fixed smoothing and bins. Target ordinary-generation outcomes were excluded from this definition. This gives reference-model-relative difficulty; mathematics uses published labels.

For 14B on MMLU-Pro, the combined-prompt gain was larger on easy items. Hard-minus-easy effect differences were −6.24, −3.62 and −2.41 points at 512, 2,048 and 8,192 tokens; all nine-contrast simultaneous bands for these differences exclude zero. Mathematics estimates also favor easier items, but bands include zero (at 8,192: −11.69 [−23.93, 0.55]). Continuous moderation and reference-label resampling check the bins.

Original gpt-oss difficulty (W2/Q2)

We matched reference difficulty to all 698 original gpt-oss items by exact question, options and correct answer. This retrospective analysis retains the original 16-token answer procedure.

On GPQA at 256 tokens, 120B high-minus-low accuracy was −0.56 points on hard items and −25.08 on easy items. The hard-minus-easy effect difference was +24.51 [3.29, 45.74] points (separate 42-contrast simultaneous 95% family). Thus high effort was relatively more favorable on hard items, although the hard-group estimate remained negative.

At higher allowances, patterns change: both models’ hard MMLU-Pro curves have two observed sign reversals. We report adjacent-allowance crossing intervals and bootstrap frequencies for no qualifying transition and multiple reversals. Original overall curves and resampling weights are unchanged.

Task differences and mathematics (W3/Q3)

For 14B at 512 tokens, the MMLU-Pro-minus-GPQA combined-prompt effect difference was 3.22 [1.00, 5.47] points (pointwise 95% interval), establishing a task difference without attributing it to difficulty.

The mathematics experiment uses the original two clauses on 500 Omni-MATH-2-Filtered items, with four generations per item and condition. Accuracy gains were 3.40, 4.50 and 9.60 points across the three allowances; all bands in the 45-contrast simultaneous family exclude zero. This addresses the earlier extension’s different prompt and single rollout. All 500 items remain included. Uncertain semantic grades receive zero primary credit, with adverse/favorable bounds; automatic delivery failures also receive zero. Grades were AI-reviewed, without human validation.

Final-answer procedures (W4/Q4)

The primary procedure regenerates final answers even after completed reasoning, allowing 64 answer tokens for multiple-choice questions and 256 for mathematics. Additional checks use a neutral answer context, valid-option-constrained generation, retained natural completions, and larger answer allowances.

On MMLU-Pro at 512 tokens, combined-prompt gains were 2.84 points in source context, 2.76 neutral, and 2.40 with valid-option generation; at 8,192 they were 0.80, 0.69 and 0.04. On the preselected 200-item mathematics subset at 8,192, increasing the final-answer allowance from 256 to 1,024 tokens reduced the gain from 12.25 to 4.88 points (paired change −7.38 [−11.75, −3.13], pointwise 95%). More answer tokens also permit computation; the check cannot isolate formatting or a procedure-independent latent belief.

We report completion and correctness separately. Conditioning on both runs remaining unfinished or on both answers parsing selects on instruction outcomes; even difficulty-adjusted comparisons are diagnostic, not causal estimates of stopping. Primary comparisons retain all items and failures in the denominator.

Thank you for the detailed rebuttal and the substantial additional experiments. I appreciate the authors' efforts to address my concerns through systematic evaluations and statistical analyses.

The new results substantially address my original concerns, particularly regarding generalization across model sizes (W1), the interaction between problem difficulty and reasoning budgets (W2), and differences across benchmarks (W3). The same-prompt comparisons, independently estimated difficulty levels, and additional mathematics experiments provide considerably stronger empirical evidence than the original submission.

I also appreciate that the authors explicitly acknowledge the limitations and inconsistencies revealed by the additional experiments. In particular, the results suggest that the effectiveness of reasoning control is highly dependent on model size, problem difficulty, reasoning budget, and evaluation procedure, rather than following a simple or universally consistent pattern.

Nevertheless, I believe a few points deserve further clarification in the revised manuscript.

Consistency of prompting conditions: Please clarify whether the "combined instructions" used in the new experiments correspond exactly to the concise prompting condition that produced the strongest results in the original paper. This is important for determining whether the additional experiments directly establish the generalizability of the original findings.

Interpretation of accuracy improvements (W4): The additional final-answer experiments are valuable. However, the results also indicate that the observed improvements can be sensitive to the final-answer generation procedure. For example, the MMLU-Pro improvement at 8,192 tokens decreases from 0.80 to 0.04 percentage points under valid-option-constrained generation. Similarly, increasing the final-answer allowance substantially reduces the observed improvement on mathematics problems. These findings suggest that improvements in measured accuracy should not necessarily be interpreted as improvements in underlying reasoning quality. I encourage the authors to emphasize this distinction and avoid causal claims that are not directly established by the experiments.

Reliability of mathematics evaluation: The additional Omni-MATH-2 experiments strengthen the paper, but the absence of human validation for AI-reviewed answer grading remains a limitation. At minimum, this limitation should be clearly acknowledged, particularly given the potential influence of answer formatting on evaluation outcomes.

Overall, I consider the authors' response convincing and believe that most of my original concerns have been substantially addressed. The additional experiments strengthen the paper's empirical contribution, although they also suggest that the main conclusions should be framed more carefully.

In my view, the paper's strongest contribution is not demonstrating that concise reasoning consistently improves accuracy, but rather providing a systematic empirical characterization of how reasoning-control strategies interact with model characteristics, problem difficulty, token budgets, and answer-generation procedures.

Reviewer_C6mzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper addresses an important question about reasoning systems, i.e. if greater control over the length of a computation translates into greater control over the quality and availability of its result. Its value extends beyond identifying a favorable prompt or effort setting. It shows that reasoning capability, resource use, and answer delivery need to be evaluated together (while remaining conceptually distinct). The empirical work is sufficiently systematic. he main reservation concerns the breadth of the scientific advance, namely the evidence establishes informative behaviors in particular systems more convincingly than general principles of adaptive reasoning.

Critical strengths
  1. The paper addresses an important question about reasoning efficiency. It distinguishes producing shorter explanations, delivering useful answers earlier, and adapting to available resources. These help to clarify what improvements in reasoning efficiency actually mean.

  2. The empirical analysis is reasonably and well qualified. The experiments include sensible controls and supporting analyses. The authors generally distinguish observed behavior from stronger claims about underlying reasoning mechanisms.

Critical weaknesses
  1. The broader explanatory contribution remains limited. The paper provides useful observations and evaluation guidance, but offers less understanding of why the studied behaviors occur or when they should be expected in other systems.

  2. The generality of the conclusions remains limited. The models and tasks support the reported case studies, but provide a limited basis for broader recommendations about efficient reasoning. The benefits also depend on whether the objective is lower cost, earlier answers, or greater eventual accuracy.

Questions for authors
  1. What is the main scientific insight about reasoning systems beyond the proposed evaluation distinctions?

  2. How does the selection of models, tasks, and interfaces support the intended scope of the conclusions?

  3. Do the findings suggest a common explanation of how models regulate reasoning, or primarily describe different behaviors?

  4. What practical guidance follows for choosing between shorter reasoning, earlier answers, and higher accuracy?

  5. Which conclusions are likely to extend to tasks where intermediate reasoning or verification is itself part of the desired output?

1 comment

Thank you for asking what the evaluation explains and how it should guide a user. The new controls identify which part of the concise instruction improves accuracy and make the practical accuracy–cost trade-off explicit.

What the controlled comparison establishes

We tested concision and answer priority separately and together in Qwen3-14B, on 1,000 MMLU-Pro, 198 GPQA Diamond and 500 open-ended mathematics items. The average concision effect improves accuracy at two MMLU-Pro allowances and all three mathematics allowances under the complete 45-contrast simultaneous family. Direct comparisons favor concision over answer priority at MMLU-Pro 512 and mathematics 2,048 and 8,192. The average answer-priority effects and interactions remain uncertain.

Completion reveals a different pattern. At 512 tokens on MMLU-Pro, adding concision increases reasoning completion by 13.30 percentage points [12.09, 14.52], while adding answer priority increases it by 2.18 [1.57, 2.79]. On mathematics, the completion changes are −0.83 [−2.40, 0.78] for concision and +3.98 [2.95, 5.03] for answer priority. These are average clause effects with pointwise 95% intervals. Thus the experiment separates effects on completion from effects on accuracy. It does not identify completion as the cause of an accuracy gain.

The accuracy comparisons use regenerated final answers, including after naturally completed reasoning, with 64 answer tokens for multiple-choice questions and 256 for mathematics. The mathematics grades are AI-reviewed, with reference corrections and ambiguity bounds, without human validation. Changing the final-answer context or allowance changes the measured gains, so the answer procedure remains part of their interpretation.

Generality across models and interfaces

We tested the identical combined instruction in Qwen3-4B and -8B on the same multiple-choice questions. On GPQA, the effect is −6.76 points for 4B at 512 tokens (simultaneous 95% band [−12.09, −1.42]) and +13.45 for 8B at 8,192 [6.05, 20.85]. These bands belong to the separate 30-contrast family. The results show where the instruction helps and where it hurts.

Using independent reference difficulty, we find larger gains from the combined prompt on easier MMLU-Pro questions for Qwen3-14B, while mathematics does not resolve a hard-versus-easy difference. Qwen3 instructions and gpt-oss effort settings provide two interface case studies. We compare policies within each model because the families and answer procedures differ.

Choosing for accuracy or average cost

For accuracy under a fixed reasoning allowance, the relevant comparison is the correctness of the delivered answer under a specified answer procedure. For lower average cost, the relevant comparison is natural-answer accuracy alongside generated tokens.

In the natural-generation comparison with a 28,000-token cap, the combined instruction lowers MMLU-Pro mean tokens from 2,327.57 to 1,649.46, with an accuracy change of −0.95 points [−1.55, −0.36]. On GPQA, mean tokens fall from 4,991.25 to 3,686.94, with an accuracy change of −2.90 points [−4.86, −0.95]. The accuracy intervals are pointwise 95% intervals. At mathematics 512, improved primary accuracy instead accompanies 36.05 additional mean path tokens [26.56, 45.64], also a pointwise interval. These are measured accuracy–token trade-offs, without a non-inferiority guarantee or a wall-clock latency estimate.

What the policy selection shows

A separate development sample selected the combined instruction at all six MMLU-Pro/mathematics task-by-allowance choices. Its primary evaluation therefore equals the fixed combined instruction by construction. The selection adds no benefit from switching policies across allowances. The protocol retains the disclosures of limited prior access to primary outputs before the choices were frozen.

The retrospective Gemma analysis likewise selects one policy per task across allowances and has substantial answer-delivery constraints. These results support an objective-specific policy comparison, rather than a question-specific compute allocator. Our accuracy endpoint evaluates final answers, not the completeness of a proof or explanation. We welcome any specific remaining question about an inference or evaluation step.

Reviewer_KwKXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how different reasoning settings affect answer accuracy under limited token budgets. It compares prompt-based reasoning controls in Qwen3 with different reasoning effort settings in gpt-oss, using shared token budgets and completion-state analysis. Experiments on GPQA Diamond and MMLU-Pro show that lower reasoning effort performs better under smaller budgets, while higher effort achieves better accuracy with larger budgets. The analysis also suggests that the early advantage of lower effort mainly comes from finishing earlier rather than consistently better intermediate reasoning. For Qwen3, concise prompting improves early answers, while specifying an exact token budget provides little additional benefit. The paper further validate their findings through additional analyses, independent replication, and limited open-ended math experiments.

Critical strengths
  1. The paper provides a clear and useful way to compare reasoning models under the same token budget. I also like the analysis of different completion states, which helps explain where the early-answer improvements come from.

  2. The paper provides some useful analyses of how different reasoning settings affect early answers.

Critical weaknesses
  1. Previous studies have already explored reasoning under limited token budgets and how answers change during reasoning [a, b]. Therefore, it is not clear what new insights it offers beyond these existing works, especially regarding why these patterns occur.

a. An Empirical Study of LLM Reasoning Ability Under Strict Output Length Constraint; https://aclanthology.org/2025.emnlp-main.389.pdf

b. Probing the Trajectories of Reasoning Traces in Large Language Models; https://arxiv.org/pdf/2601.23163

  1. The experiments are mainly conducted on a few models and multiple-choice benchmarks, so it is unclear whether the findings generalize to other tasks. And some improvements may come from answer formatting rather than better reasoning.
Questions for authors
  1. The concise prompt asks the model to both reason concisely and reach an answer quickly. Could you investigate these two instructions separately to better understand which one contributes more to the improvement?
  2. Could you further investigate whether the early-answer improvements also hold for more open-ended tasks, such as mathematical reasoning or code generation, rather than mainly multiple-choice questions?
1 comment

Thank you for pointing out that the original instruction combines two requests. We have now tested each request separately and repeated the same intervention on open-ended mathematics.

Separating concision from answer priority

We crossed “Use a very concise reasoning strategy” with “Prioritize reaching a usable answer quickly” in Qwen3-14B. This gives four conditions: ordinary generation, concision alone, answer priority alone, and both requests together. The study covers 1,000 MMLU-Pro, 198 GPQA Diamond and 500 Omni-MATH-2-Filtered items, with eight generations per item and condition for multiple-choice questions and four for mathematics. Each condition uses reasoning allowances of 512, 2,048 and 8,192 tokens. Our primary procedure regenerates a final answer from each reasoning prefix, including naturally completed reasoning, with 64 answer tokens for multiple-choice questions and 256 for mathematics.

To estimate the average concision effect, we compare adding concision with and without answer priority, then average those two comparisons. At 512 reasoning tokens, accuracy improves by 2.46 percentage points on MMLU-Pro (simultaneous 95% band [1.17, 3.76]) and 2.90 on mathematics [0.45, 5.35]. The effect is also positive at MMLU-Pro 2,048 and all three mathematics allowances, reaching 7.32 points [3.87, 10.78] at 8,192.

The direct comparison between the two single-clause prompts favors concision at MMLU-Pro 512 by 2.09 points [0.68, 3.49], and at mathematics 2,048 and 8,192 by 5.05 points in each case, with bands [1.79, 8.31] and [1.40, 8.70]. These bands cover the complete family of 45 accuracy contrasts. The average answer-priority effects, interactions and GPQA accuracy contrasts remain unresolved under this family.

The same prompt on open-ended mathematics

The combined instruction improves mathematics accuracy over ordinary generation by 3.40, 4.50 and 9.60 points at 512, 2,048 and 8,192 reasoning tokens. The corresponding simultaneous bands are [0.40, 6.40], [1.19, 7.81] and [4.91, 14.29]. This tests the original two clauses with repeated generations, addressing the earlier extension’s different prompt and single rollout.

All 500 items were retained in the analysis. The grades use a frozen judge followed by blinded AI review and reference adjudication. Five reference corrections apply across every affected policy and answer procedure. The under-specified item retains ambiguous semantic grades with adverse and favorable bounds, and automatic delivery failures retain zero credit. The grading amendments reduced the 8,192-token gain by 0.40 points. These bounds cover identified ambiguity, and the new grades have no human validation.

How the answer procedure changes the gain

Using the same neutral final-answer context for both policies gives mathematics gains of 1.70, 2.15 and 7.20 points. Relative to the primary procedure, the paired changes are −1.70 [−3.20, −0.20], −2.35 [−3.60, −1.15] and −2.40 [−3.60, −1.25], with pointwise 95% intervals.

On the preselected 200-item mathematics subset at 8,192 reasoning tokens, increasing the answer allowance from 256 to 1,024 tokens changes the gain from 12.25 to 4.88 points. The paired change is −7.38 [−11.75, −3.13], again with a pointwise interval. The larger allowance permits additional computation, so this comparison cannot isolate formatting. The full-cohort common-context and multiple-choice valid-option checks are reported alongside these results.

Contribution and reasoning interfaces

We build on Sun et al.’s strict-budget evaluation and Ballon et al.’s trajectory probing. The new experiment identifies which instruction component improves answer accuracy and shows that completion and accuracy respond differently. For example, answer priority increases mathematics completion at 512 tokens, while its average accuracy effect remains uncertain.

Qwen3 prompt instructions and gpt-oss effort settings provide two interface case studies, with policy comparisons within each model. Their different model families and answer procedures prevent a causal comparison of the interfaces. The earlier announced-budget tests separately examine adaptation to the stated allowance. Our contribution concerns these observable changes in answer behavior; the completion analyses do not identify an internal mechanism or the causal effect of stopping alone.

Comments on the paper

1 comment

Thank you for the careful reviews. We have separated the two requests in our concise prompt, tested the same wording across tasks and model sizes, and added an independent measure of question difficulty. The results clarify when concise instructions improve answers and why reasoning completion needs to be measured separately.

Contribution and scope

We study how reasoning controls change the answer available at a given reasoning allowance. Prompt instructions in Qwen3 and effort settings in gpt-oss provide two interface case studies. We compare policies within each model, since the model families and answer procedures differ.

We build on Sun et al.’s strict-budget evaluation and Ballon et al.’s trajectory probing. Our new factorial identifies accuracy gains from the concision clause on MMLU-Pro and open-ended mathematics. The completion and answer-procedure analyses then clarify how to interpret those gains. Together with the earlier announced-budget tests, this separates improved early answers, earlier completion and adaptation to budget information.

Separating concision from answer priority

Our original prompt requested concise reasoning and a usable answer quickly. We tested each request alone and both together against ordinary generation in Qwen3-14B. The comparison covers 1,000 MMLU-Pro, 198 GPQA Diamond and 500 Omni-MATH-2-Filtered items, with eight generations per item and condition for multiple-choice questions and four for mathematics. Our primary procedure generates a fresh final answer from each reasoning prefix, including naturally completed reasoning, with 64 answer tokens for multiple-choice questions and 256 for mathematics.

At 512 reasoning tokens, adding concision improves accuracy by 2.46 percentage points on MMLU-Pro (simultaneous 95% band [1.17, 3.76]) and 2.90 on mathematics [0.45, 5.35], averaging over whether answer priority is present. Directly comparing the two single-clause prompts, concision is 2.09 points more accurate on MMLU-Pro [0.68, 3.49]. These bands cover the full family of 45 accuracy contrasts. The average answer-priority effect and the interaction remain uncertain, as do the GPQA accuracy contrasts.

On mathematics at 512 tokens, answer priority increases reasoning completion, while its average accuracy effect remains uncertain. This is why we report completion and correctness separately rather than treating earlier completion as an explanation of the accuracy gain.

Mathematics and the answer procedure

The original two-clause prompt improves mathematics accuracy by 9.60 points at 8,192 reasoning tokens [4.91, 14.29], under the same 45-contrast family and the 256-token regenerated-answer procedure above.

Giving both policies the same neutral final-answer context yields a 7.20-point gain on the same 500 problems. The paired reduction is 2.40 points (pointwise 95% interval [1.25, 3.60]). On the preselected 200-item mathematics subset at 8,192 reasoning tokens, increasing the answer allowance from 256 to 1,024 tokens changes the gain from 12.25 to 4.88 points. The larger allowance permits additional computation. The gain persists with a neutral context, but its magnitude also depends on the computation allowed for the final answer.

All 500 mathematics items were retained in the analysis. The grading amendments reduced the 8,192-token gain by 0.40 points. We report adverse and favorable bounds for uncertain semantic grades. The new mathematics grades are AI-reviewed, without human validation.

Where the gains hold

We tested the identical two-clause prompt in Qwen3-4B and -8B on the same multiple-choice questions. On GPQA, the effect is −6.76 points for 4B at 512 tokens [−12.09, −1.42] and +13.45 for 8B at 8,192 [6.05, 20.85], using 30-contrast simultaneous 95% bands. The gains therefore depend on the model, task and allowance.

Difficulty now uses 16 separate ordinary Qwen3-14B answers per multiple-choice question and published labels for mathematics. For 14B, the gain from the combined prompt is larger on easier MMLU-Pro questions across all three allowances. Mathematics does not resolve a hard-versus-easy difference. The additional gpt-oss difficulty analysis is retrospective and includes multiple curve crossings.

Practical implications

Accuracy at a fixed allowance and lower average token use are different objectives. In the natural-generation comparison with a 28,000-token cap, the two-clause prompt uses fewer tokens, with a −0.95-point change in MMLU-Pro accuracy (pointwise 95% interval [−1.55, −0.36]). The earlier announced-budget tests also found no clear additional accuracy benefit from matching the announced number to the imposed stop. These comparisons give a more specific basis for choosing a reasoning policy than trace length alone.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.