- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates the impact of fixed length penalties on accuracy and reasoning efficiency in reinforcement learning for mathematical reasoning. The authors introduce a normalized and clipped length penalty into the correctness reward, without requiring additional difficulty estimation or dynamic parameter adjustment. Using DeepSeek-R1-Distill-Qwen-1.5B, the paper compares the proposed method against several adaptive length control approaches across four mathematical reasoning benchmarks. The results show that the method reduces generated tokens by 2.8%–26.9% while largely maintaining accuracy, suggesting that simple fixed penalties can serve as an effective baseline for evaluating more sophisticated length control methods.
Critical strengths
The proposed method is simple and easy to implement, requiring neither additional difficulty estimation nor dynamic reward adjustment mechanisms, which gives it practical value. Experimentally, the authors compare their approach with several adaptive length control methods across four mathematical reasoning benchmarks and evaluate the stability of the results using multiple random seeds. In addition, the paper analyzes the effects of different penalty coefficients, problem difficulty levels, and model scales on reasoning compression, providing empirical insights into both the effectiveness and limitations of fixed length penalties.
Critical weaknesses
W1. Limited methodological novelty and substantial overlap with prior work.
This paper reduces reasoning token consumption by introducing a fixed length penalty into the correctness reward. However, similar ideas have already been explored in prior studies, such as Arora & Zanette (2025) and L1 (2025). In particular, LEASH (arXiv:2512.21540) has already introduced a fixed-penalty variant, LEASH-C, and compared the effectiveness of fixed and adaptive length penalties.
Therefore, although this paper provides some empirical value, its core idea and main findings are closely related to existing research. The authors should further clarify the unique contributions of this work and provide more comprehensive discussions and comparisons with directly relevant studies.
Relevant references:
- Arora & Zanette (2025). Training Language Models to Reason Efficiently
- Aggarwal & Welleck (2025). L1: Controlling How Long a Reasoning Model Thinks With Reinforcement Learning
- LEASH (2025). Adaptive Length Penalty and Reward Shaping for Efficient Large Reasoning Model
W2. Limited model scale and outdated evaluation benchmarks.
The main experiments are conducted only on DeepSeek-R1-Distill-Qwen-1.5B, with reinforcement learning training data primarily drawn from GSM8K. Moreover, the evaluation benchmarks are relatively outdated, especially considering that newer benchmarks, such as AIME 2026, are now available.
Questions for authors
see above
3 comments
Thank you sincerely for asking us to clarify what this study adds relative to LEASH-C and other established length-control work. Following your suggestions, we completed additional cross-model training and expanded the empirical questions. The added content concerns three areas.
First, we broadened model coverage and tested which observations transfer. We completed Correctness/Fixed × group-SD/no-SD training on Qwen2.5-Math-1.5B-Instruct and DeepSeek-R1-Distill-Llama-8B, with 400 updates per arm, four mathematical benchmarks, and two total-context budgets per model. This adds evidence from a Llama backbone. For example, Qwen/MATH/4K/group saves 4.26% of tokens, and Llama/MATH/8K/no-SD saves 6.38%. We also retain Llama/group’s quality deterioration and the changes in difficulty profiles and tail-compression relationships across the new models. These outcomes help delineate which observations transfer and which depend on the configuration.
Second, we examined whether compression training remains compatible with inference-time answer selection. In the original eight-seed GSM8K/group cohort with a 3,072-token completion cap, the advantage of shortest over random selection is +2.93 percentage points for Correctness but −2.46 points for Fixed; every seed exhibits this sign reversal. Five-fold question-held-out calibration on the same Fixed answer pool achieves 83.98% selection accuracy, compared with 81.99% when transferring ranks calibrated on Correctness outputs. Equally calibrated Correctness remains higher at 87.04%, so the additional finding concerns selection-rule adaptation. Calibration uses correctness labels only on its training folds, and all generation costs have already been incurred. The new models provide local support and counterexamples, showing that failure of shortest selection should not be treated as universal.
Third, we examined how saved tokens translate into finite-sampling utility. At equal expected generation-token cost, oracle coverage in the original eight-seed GSM8K/group cohort increases from 92.69% to 93.95%. The new models also provide local gains—for example, +0.69 percentage points for Llama/no-SD on GSM8K at 32K—alongside counterexamples such as −0.62 points on MATH-500 at 32K. This measures whether at least one correct answer is generated and still requires a verifier. It addresses whether cheaper responses can support greater problem coverage under a matched expected budget.
These extensions develop the empirical content around model transfer, compatibility between training objectives and inference-time selection, and sampling-budget reallocation. We fully recognize precedents for fixed penalties, length selection, calibration and pass@k; the added value lies in the specific controlled evidence and applicability boundaries. The new model runs remain single-training-seed GSM8K experiments, and the extended analyses are post hoc. Together with the previously reported AIME2026 evaluation, they supplement the original eight-seed results. Thank you for helping us make the empirical contribution more concrete and testable.
Response to rebuttal
Thank you for the detailed response. The authors have generalized the model to 7B and also tested it on AIME26. However, looking at the overall results, the method performs significantly better on the relatively simpler MATH dataset, while its impact on more challenging datasets appears less pronounced. I will maintain my current score.
Thank you for highlighting the connection to LEASH-C and for suggesting broader evaluation across models, tasks, and benchmarks. Your feedback helped us sharpen the empirical contribution: eight-seed controls identify how normalization and decoding budgets shape accuracy–length trade-offs, while direct prior-method comparisons and the 7B, AIME2026, and local MBPP extensions broaden the evidence. Together, these results clarify the conditions under which a simple fixed length cost reduces token use.
We agree that fixed length penalties and fixed-versus-adaptive comparisons have precedents. The Introduction and Related Work now acknowledge LEASH-C explicitly. Arora and Zanette study length-aware RL; L1 learns prompt-specified length constraints; LEASH uses a target-excess cost with dual adaptation, and LEASH-C fixes its coefficient. Our capped completion cost charges length from the first token, including incorrect responses. This reward difference defines the experiment, not priority for the general principle.
The added controls supply the empirical contribution. An eight-training-seed Correctness/Fixed × group-SD/no-SD factorial gives MATH/AIME/AMC token savings of 14.64%/9.05%/10.85% with scaling versus 3.31%/2.12%/5.10% without at 32K (Table 10). Five of six token contrasts, but none of the accuracy contrasts, pass the corresponding Holm correction. The lambda × SD sweep shows further normalization dependence (.2/group six seeds; other cells eight). Saved-checkpoint long-budget evaluation shows that short-budget accuracy advantages disappear or reverse while token savings persist (Appendix I). These findings specify the conditions under which a fixed reference is informative; they do not establish accuracy equivalence or universal superiority.
Appendix J adds Arora's released-recipe RLOO comparison (one training seed, 100 rollout rounds/200 Adam updates per arm) and controlled LEASH/LEASH-C adaptations (two seeds/400 updates). The latter are not full original-compute reproductions. Arora's asymmetric recovery histories and unrestored private generation RNG further limit method ranking; unfavorable comparisons are retained. The finite seven-configuration, two-seed tuning grid yields no candidates satisfying its specified accuracy constraints, so there is no selected independent-test Pareto frontier. We acknowledge that a complete tuned comparison and broader methodological novelty are not established by these results.
Healthy DeepSeek-R1-Distill-Qwen-7B runs use two seeds per arm, BF16 full-parameter training, no group-SD, and a matched initialization (Appendix L, Table 15). At 32K, Fixed saves 4.31%/0.86%/3.19% tokens on MATH/AIME/AMC, with accuracy changes of +0.013/−0.069/+1.280 pp. Savings vary across tasks. A comparison with the primary 1.5B group-SD results changes both size and normalization and does not isolate a scale effect; at matched no-SD, MATH savings are 3.31% for 1.5B versus 4.31% for 7B. The old approximately 2% anomaly predates RL and is retained only as diagnostic background. Two seeds and one model family do not establish broad model generalization.
All 30 AIME2026 questions are evaluated with eight paired training seeds and eight responses per question at 8K completion and 32K total context (Appendix M, Table 26). Accuracy changes are −0.677/+0.729 pp, with token savings of 0.76%/0.73%. Neither metric passes its two-budget Holm correction. This adds current benchmark coverage, with small uncertain effects; 30 questions and multiple samples do not establish strict decontamination or population-wide performance. The initialization and complete results are included.
The second training-domain check uses MBPP: 120 training problems, 100 updates, one seed, and 64 disjoint test problems × four responses (Appendix N). Correctness/Gated/Ungated yield 125/126/0 successes out of 256 at about 686/349/4 tokens. Gating is worth investigating locally, but this is not a full multi-seed non-math study and does not erase historical gated HumanEval degradation or ungated collapse. Primary training still uses one prompt × eight responses/update for about 400 distinct problems/run; data/compute scaling and independent model families remain outside the completed scope.
The revision therefore contributes a controlled empirical reference with measured optimization, budget, and task boundaries, rather than a claim that fixed penalties are new or reliably outperform adaptive methods.