W1. Limited generalizability across model architectures and scales
The paper provides insufficient evidence that its main findings generalize consistently across different model architectures and sizes.
Although the experiments include Qwen3-4B/8B/14B and gpt-oss-20b/120b, the main prompt-based reasoning control experiments primarily focus on Qwen3-14B. Similarly, the reasoning effort experiments are restricted to two models from the same gpt-oss family.
Appendix Table A31 presents additional results for Qwen3-8B, but these experiments do not demonstrate consistent accuracy improvements under the Numeric/Concision instruction. However, the evaluated instruction differs from the concise prompt that produces the strongest results for Qwen3-14B. Therefore, these experiments do not directly establish whether the main concise-prompt effect generalizes across model sizes.
Furthermore, Qwen3 and gpt-oss are evaluated using different reasoning control mechanisms. Consequently, it is difficult to separate the effects of model architecture, training, and reasoning control interfaces.
While the reported results provide useful empirical observations for the evaluated settings, the evidence remains insufficient to support broader conclusions about reasoning behavior across LLMs.
I suggest evaluating the same concise reasoning instruction across multiple Qwen3 model sizes and, if possible, examining whether similar effects can be reproduced in additional model families under comparable reasoning control settings.
W2. Lack of systematic analysis of problem difficulty and its interaction with reasoning budgets
A major limitation is the absence of a systematic analysis of how problem difficulty affects the effectiveness of reasoning control.
The paper demonstrates that lower reasoning effort or concise prompting can be advantageous under short reasoning budgets, while these advantages diminish or reverse as the reasoning budget increases.
However, this relationship may depend strongly on problem difficulty.
For relatively easy questions, concise reasoning may be sufficient to reach the correct answer. In contrast, complex problems requiring multiple reasoning steps may benefit substantially from longer reasoning trajectories, and prematurely terminating the reasoning process could reduce accuracy.
Although the authors use difficulty-stratified sampling for Omni-MATH-2, they do not report performance separately across difficulty levels. Similarly, Appendix Table A32 presents category-level accuracy differences for MMLU-Pro, but subject categories should not be treated as equivalent to problem difficulty.
This issue is particularly relevant to the completion-state analysis. If easier questions tend to be completed earlier, conditioning on both policies remaining unfinished may introduce selection bias, potentially changing the difficulty distribution of the evaluated subset.
The authors acknowledge this selection issue, but do not explicitly control for problem difficulty.
I believe the following analyses would substantially strengthen the paper:
Divide benchmark questions into difficulty groups (e.g., Easy, Medium, and Hard) using an independent difficulty measure.
Compare accuracy and completion rates across reasoning budgets within each difficulty group.
Examine whether the relative advantages of Low/High effort or Concise/Ordinary prompting change significantly with problem difficulty.
Investigate whether the reasoning budget at which high effort overtakes low effort differs across difficulty levels.
Such analyses would provide a more informative explanation of when and why shorter reasoning is beneficial, rather than only demonstrating that shorter reasoning can improve early-answer accuracy in certain evaluation settings.
W3. Inconsistent effects across benchmarks and limited cross-task validation
The effectiveness of reasoning control does not appear to be equally consistent across the evaluated benchmarks.
For example, Appendix Table A32(b) reports that the concise instruction improves Qwen3-14B accuracy by approximately 4.9 percentage points on MMLU-Pro at a 512-token budget, whereas the corresponding improvement on GPQA Diamond is only approximately 0.2 percentage points.
Although these results do not establish a statistically significant difference between the two benchmarks, they suggest that the effectiveness of concise reasoning may depend on task characteristics.
The Omni-MATH-2 experiments are a useful extension beyond multiple-choice questions. However, the Qwen3 experiments on this benchmark evaluate the Numeric/Concision instruction rather than the concise instruction that yields the strongest main results. Moreover, these experiments use only a single rollout per condition.
Consequently, the additional benchmark results provide limited evidence that the main Qwen3 findings generalize to different reasoning tasks.
Given the paper's broader motivation of evaluating reasoning efficiency, a more systematic analysis across benchmarks would improve the strength of its conclusions.
In particular, it would be useful to investigate whether differences across benchmarks can be explained by problem difficulty, task structure, or the amount of reasoning required to obtain a correct answer.
W4. Interpretation of early accuracy improvements
The interpretation of early-answer accuracy improvements also deserves further clarification.
The authors appropriately distinguish earlier completion from improvements in unfinished reasoning quality and acknowledge that the completion-state decomposition does not establish a causal stopping-only mechanism.
Nevertheless, some of the observed improvements in the Qwen experiments are partly attributable to differences in answer parseability.
This raises an important question regarding the extent to which early-answer improvements reflect better information acquisition during reasoning, changes in reasoning termination behavior, or improvements in final-answer generation and formatting.
A clearer discussion or additional analysis separating these factors would strengthen the interpretation of the experimental results.
Thank you for asking for a clearer account of what the evaluation teaches us. We have reorganized the paper around the clause experiment and the finding it supports.
What the clause comparison shows
The new Qwen3-14B study tests concision and answer priority separately and together on 1,000 MMLU-Pro, 198 GPQA Diamond and 500 open-ended mathematics items. At 512 reasoning tokens, adding concision improves accuracy by 2.46 percentage points on MMLU-Pro (simultaneous 95% band [1.17, 3.76]) and 2.90 on mathematics [0.45, 5.35], averaging over whether answer priority is present.
The average concision effect is positive at two MMLU-Pro allowances and all three mathematics allowances under the complete 45-contrast family. Direct comparisons between the single-clause prompts favor concision at MMLU-Pro 512 and mathematics 2,048 and 8,192. The average answer-priority effects and interactions remain uncertain.
Why completion and the answer procedure matter
On mathematics at 512 tokens, answer priority increases reasoning completion by 3.98 points [2.95, 5.03], while concision changes it by −0.83 [−2.40, 0.78]. These are average clause effects with pointwise 95% intervals. Completion and accuracy therefore give different information about the instruction. This is evidence about observable behavior, not an internal mechanism or a causal effect of stopping alone.
The primary accuracy procedure regenerates final answers from reasoning prefixes, including naturally completed reasoning, with 64 answer tokens for multiple-choice questions and 256 for mathematics. Changing the final-answer context or allowance changes the measured prompt gain. The new mathematics grades are AI-reviewed, with corrected references and bounds for identified ambiguity, without human validation.
Contribution and clearer presentation
The controlled 14B experiment identifies accuracy gains from the concision clause. Separate checks of the original two-clause prompt show how its benefit varies across models, tasks and answer procedures. The identical combined prompt produces both gains and losses across Qwen sizes. Lower natural token use also accompanies accuracy losses on MMLU-Pro and GPQA. With independently measured reference difficulty, we find larger gains from the combined prompt on easier MMLU-Pro questions for Qwen3-14B, while the mathematics difficulty comparison remains unresolved. The earlier announced-budget matching tests remain separate and leave room for small matching benefits.
We build on Sun et al.’s strict-budget evaluation and Ballon et al.’s trajectory probing. Qwen3 prompt instructions and gpt-oss effort settings provide two interface case studies, with controlled policy comparisons within each model and different families and answer procedures across interfaces.
The revised main text introduces the question and primary answer procedure, then presents the clause comparison, model/task scope and natural accuracy–cost trade-offs. Earlier matching, completion-state and answer-probability analyses remain in the historical appendix. This gives the reader the finding and the evidence needed to interpret it before the detailed diagnostics.