- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies the length-scaling tax (LST): increased response length on already-solved queries during reinforcement-learning post-training without a corresponding accuracy gain. The authors measure this phenomenon using fixed easy-query sets and a shared, accuracy-qualified reference length from RL checkpoints. To mitigate LST, the paper proposes Length Self-Distillation (LSD). Fully correct rollout groups receive on-policy distillation, while the remaining groups retain the original RL objective. Experiments compare three distillation variants on single-turn mathematical reasoning and multi-turn agentic tasks, showing reduced length growth on easy queries with comparable or improved overall performance.
Critical strengths
-
Meaningful phenomenon and controlled evaluation.
Tracking fixed easy-query cohorts and reporting accuracy alongside length help isolate unnecessary length growth and distinguish effective compression from capability loss. -
Clear motivation and empirical support.
LSD supplies token-level supervision to all-correct groups lacking relative reward signals while retaining RL elsewhere. Results across mathematical reasoning and agentic tasks support its ability to preserve concise responses on easy queries while maintaining overall performance.
Critical weaknesses
-
Comparisons with relevant strong baselines are incomplete.
External efficiency-method comparisons primarily include CRISP for mathematical reasoning and an AdapThink-inspired length penalty for agentic tasks. Other related approaches discussed in the paper, including DAST and Contrastive On-Policy Distillation, as well as additional newer baselines, are not included in the experiments. The current comparisons therefore do not sufficiently establish LSD’s advantage over a broader range of difficulty-aware or distillation-based reasoning-efficiency methods. Additional representative baselines should be evaluated with the same backbone, tasks, and training conditions, comparing the trade-off between accuracy and response length. -
The potential entropy–compression trade-off and its consequences for continued training are not sufficiently examined.
The paper does not report the final main method’s policy entropy relative to the initial model and the standard RL baseline under identical evaluation conditions. The appendix reports average entropy for different LSD variants over part of training, which does not establish whether SG-FKL’s compression on easy queries is accompanied by reduced policy entropy or generation diversity. If compression restricts exploration diversity, it could limit the discovery of new solution paths and subsequent capability acquisition during continued training. The paper should compare entropy and generation diversity before and after training, particularly on identical easy-query sets, and evaluate longer continued-training trajectories to assess this potential negative effect. -
Statistical robustness and generalizability remain insufficiently established.
The mathematical evaluation includes only 143 queries, with 23 in the strict LST1000 easy cohort. Pass@1 gains of 1.30 and 1.22 percentage points on mathematical and agentic tasks lack uncertainty intervals, leaving their stability unclear. Selecting the minimum length across accuracy-qualified checkpoints as the LST reference also warrants sensitivity analysis for sampling noise and evaluation frequency. Experiments cover only Qwen3 models at 4B and 8B scales; validation across other model families and sizes is needed.
Questions for authors
See Critical Weaknesses.
3 comments
Part II of the Authors' Response
W3: Statistical robustness and generalizability remain insufficiently established.
Thank you for the suggestion. We add 20k paired bootstrap resamples at the question level, using identical sampled queries for GRPO and LSD to compute 95% confidence intervals. Appendix G.2, Figure 25 presents results on 22 fixed easy queries, with 32 responses per query per method. Relative to GRPO, SG-RKL and PG-RKL reduce response length by 24.05% and 19.59%, respectively, indicating robust length reductions on this fixed easy-query set.
| Method on 22 fixed easy queries | Length reduction vs. GRPO, % [95% CI] | Accuracy difference vs. GRPO, percentage points [95% CI] |
|---|---|---|
| LSD (SG-RKL) | 24.05 [18.19, 30.07] | +1.28 [+0.43, +2.27] |
| LSD (PG-RKL) | 19.59 [13.67, 25.41] | +0.99 [−0.14, +2.13] |
Regarding your concern about the statistical reliability of the small Pass@1 gains, LSD’s primary contribution is to control response-length growth on solved queries. We do not base our main conclusions on the small Pass@1 gains in the mathematics and agentic tasks. In the additional Countdown experiment (Appendix B, Figures 15 and 16), we evaluate 384 fixed test queries with 32 responses per query. LSD improves overall Pass@1 over GRPO by 3.82 points while reducing average response length by 17.28%. On the same 326 queries with at least eight correct responses under both methods, the expected number of distinct canonical expression trees among eight correct responses is 2.222 for LSD and 2.100 for GRPO. Thus, in this experiment, LSD produces shorter responses while achieving higher success rates and higher structural diversity of correct solutions.
For the LST reference, switching to another stored checkpoint satisfying the accuracy constraint changes each method’s LST by less than 0.091 points.
| Method | LST with stored reference step 885 (%) | LST with stored reference step 1000 (%) | Absolute change (percentage points) |
|---|---|---|---|
| GRPO | 19.0301 | 18.9397 | 0.0905 |
| LSD (SG-FKL) | −3.7151 | −3.7882 | 0.0732 |
| LSD (SG-RKL) | −10.9162 | −10.9839 | 0.0677 |
| LSD (PG-RKL) | 1.4459 | 1.3688 | 0.0771 |
Thanks for the attentive reading of the manuscript and constructive feedback. We have incorporated these changes into our revised version. We hope our response addresses all the concerns and that the reviewer will consider raising the rating accordingly. We are more than glad to answer any further questions.
Dear Reviewer,
As the author-reviewer cycle nears its conclusion, we noticed that we have not yet received your response to our rebuttal. We invested considerable effort in comprehensively addressing each of your concerns, including conducting additional experiments, clarifying technical details and methodological aspects of the paper, and revising the manuscript accordingly.
We would be deeply grateful if you could review our detailed rebuttal and consider whether the substantial improvements and clarifications we have provided address your initial concerns. Your feedback is invaluable to us, and we believe our responses demonstrate significant enhancements to the work's quality and rigor.
Given the approaching deadline, we respectfully request your reconsideration of the manuscript in light of these updates. We sincerely appreciate your time and expertise throughout this process.
Thank you for your review, and we always look forward to your response.
Part I of the Authors' Response
Thank you very much for recognizing our controlled evaluation of LST and the motivation and empirical support for LSD. We appreciate your detailed feedback and address each concern below.
Thanks for your suggestion. To broaden the comparison with LSD, we add DAST and LCPO as representative baselines and evaluate three length-control strengths for each. The comparison uses Qwen3-4B-Base, the same mathematics training data and evaluation queries. For DAST, we vary the length coefficient in the correct-response reward over a = 0.5, 1.0, and 2.0; for LCPO, we test α = 3×10⁻⁵, 10⁻⁴, and 3×10⁻⁴. Other settings are held fixed within each method to assess sensitivity to length-control strength.
Results are reported in Appendix C, Figure 17, with the full LCPO training curves in Section 5.5, Figure 10. Within the tested parameter ranges, both DAST and LCPO remain below LSD in accuracy despite varying the length-penalty strength. We also discuss potential reasons for DAST’s lower performance in the appendix.
Thank you for drawing attention to the potential effects of compression on entropy and continued learning. To examine the potential trade-off between entropy and compression, we add training-entropy curves for GRPO and the three LSD variants in Appendix F.1, Figure 21. LSD (SG-FKL) maintains relatively stable training entropy above GRPO even after more than 1,600 training steps.
To examine whether distillation on solved queries restricts subsequent exploration, we add a Countdown combinatorial-search experiment (Appendix B, Figures 15 and 16). This task requires selecting number orderings, operators, and parentheses for multi-step solutions. Each query can admit multiple valid solutions, and correctness can be verified exactly, allowing us to examine success rate, response length, and the structural diversity of correct solutions. We evaluate on 384 fixed test queries, sampling 32 responses per query for each method.
At the final checkpoint, LSD achieves 82.16% overall accuracy, compared with GRPO’s 78.34%, while reducing average response length by 17.28%. On the fixed set of 200 initially hard queries, Pass@1 increases from 61.28% with GRPO to 66.42% with LSD, and Pass@32 increases from 89% to 94%. On the same 326 queries with at least eight correct responses under both methods, we compute the expected number of distinct canonical expression trees among eight correct responses. This value is 2.222 for LSD and 2.100 for GRPO. It shows that LSD improves success on initially hard queries while achieving higher structural diversity.
Figure 16 provides a concrete example: both methods answer all 32 samples correctly, but LSD produces six correct expression structures compared with GRPO’s three, while reducing average response length by 36.47%.