acceptodds
Under review as a conference paper at ICLR 2027

Mitigating the Length-Scaling Tax with Online Distillation

Abstract

Length scaling during reinforcement-learning (RL) post-training is often viewed as a sign of improved reasoning ability, especially on difficult problems, but may also make responses to already-solved problems unnecessarily verbose. We quantify this side effect as the length-scaling tax (LST): excess response length on already-solved queries without a commensurate accuracy gain. To mitigate LST, we propose Length Self-Distillation (LSD), which routes solved prompts to on-policy distillation and retains the original RL objective for unsolved prompts. LSD uses an exponential moving average of the online policy as its teacher, requiring no external model. We find that LSD achieves comparable or better performance than RL across multiple variants, while substantially curbing response-length growth on easy queries. LSD reduces LST from 19.0% to −3.7% on single-turn reasoning and from 31.4% to 13.7% on multi-turn agentic tasks, demonstrating that LSD effectively preserves concise response patterns on easy queries while supporting efficient exploration on difficult queries during RL post-training.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Length Scaling, On-policy Distillation, Large Language Model; Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_F19WOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the length-scaling tax (LST): increased response length on already-solved queries during reinforcement-learning post-training without a corresponding accuracy gain. The authors measure this phenomenon using fixed easy-query sets and a shared, accuracy-qualified reference length from RL checkpoints. To mitigate LST, the paper proposes Length Self-Distillation (LSD). Fully correct rollout groups receive on-policy distillation, while the remaining groups retain the original RL objective. Experiments compare three distillation variants on single-turn mathematical reasoning and multi-turn agentic tasks, showing reduced length growth on easy queries with comparable or improved overall performance.

Critical strengths
  1. Meaningful phenomenon and controlled evaluation.
    Tracking fixed easy-query cohorts and reporting accuracy alongside length help isolate unnecessary length growth and distinguish effective compression from capability loss.

  2. Clear motivation and empirical support.
    LSD supplies token-level supervision to all-correct groups lacking relative reward signals while retaining RL elsewhere. Results across mathematical reasoning and agentic tasks support its ability to preserve concise responses on easy queries while maintaining overall performance.

Critical weaknesses
  1. Comparisons with relevant strong baselines are incomplete.
    External efficiency-method comparisons primarily include CRISP for mathematical reasoning and an AdapThink-inspired length penalty for agentic tasks. Other related approaches discussed in the paper, including DAST and Contrastive On-Policy Distillation, as well as additional newer baselines, are not included in the experiments. The current comparisons therefore do not sufficiently establish LSD’s advantage over a broader range of difficulty-aware or distillation-based reasoning-efficiency methods. Additional representative baselines should be evaluated with the same backbone, tasks, and training conditions, comparing the trade-off between accuracy and response length.

  2. The potential entropy–compression trade-off and its consequences for continued training are not sufficiently examined.
    The paper does not report the final main method’s policy entropy relative to the initial model and the standard RL baseline under identical evaluation conditions. The appendix reports average entropy for different LSD variants over part of training, which does not establish whether SG-FKL’s compression on easy queries is accompanied by reduced policy entropy or generation diversity. If compression restricts exploration diversity, it could limit the discovery of new solution paths and subsequent capability acquisition during continued training. The paper should compare entropy and generation diversity before and after training, particularly on identical easy-query sets, and evaluate longer continued-training trajectories to assess this potential negative effect.

  3. Statistical robustness and generalizability remain insufficiently established.
    The mathematical evaluation includes only 143 queries, with 23 in the strict LST1000 easy cohort. Pass@1 gains of 1.30 and 1.22 percentage points on mathematical and agentic tasks lack uncertainty intervals, leaving their stability unclear. Selecting the minimum length across accuracy-qualified checkpoints as the LST reference also warrants sensitivity analysis for sampling noise and evaluation frequency. Experiments cover only Qwen3 models at 4B and 8B scales; validation across other model families and sizes is needed.

Questions for authors

See Critical Weaknesses.

3 comments
AuthorsOpenReview ↗

Part I of the Authors' Response

Thank you very much for recognizing our controlled evaluation of LST and the motivation and empirical support for LSD. We appreciate your detailed feedback and address each concern below.

W1: Comparisons with relevant strong baselines are incomplete.

Thanks for your suggestion. To broaden the comparison with LSD, we add DAST and LCPO as representative baselines and evaluate three length-control strengths for each. The comparison uses Qwen3-4B-Base, the same mathematics training data and evaluation queries. For DAST, we vary the length coefficient in the correct-response reward over a = 0.5, 1.0, and 2.0; for LCPO, we test α = 3×10⁻⁵, 10⁻⁴, and 3×10⁻⁴. Other settings are held fixed within each method to assess sensitivity to length-control strength.

Results are reported in Appendix C, Figure 17, with the full LCPO training curves in Section 5.5, Figure 10. Within the tested parameter ranges, both DAST and LCPO remain below LSD in accuracy despite varying the length-penalty strength. We also discuss potential reasons for DAST’s lower performance in the appendix.

MethodLength-control strengthTraining updatesOverall Pass@1 (%)Mean response length (tokens)
DASTa = 0.5184 (offline)12.39272.28
DASTa = 1.0184 (offline)11.25302.08
DASTa = 2.0184 (offline)11.32303.24
LCPOα = 3×10⁻⁵60030.33887.76
LCPOα = 10⁻⁴60032.361,127.34
LCPOα = 3×10⁻⁴60030.44788.76
LSD (SG-FKL)—60037.222,053.80
LSD (SG-RKL)—60034.071,848.02
LSD (PG-RKL)—60035.582,044.65

W2: The potential entropy–compression trade-off and its consequences for continued training are not sufficiently examined.

Thank you for drawing attention to the potential effects of compression on entropy and continued learning. To examine the potential trade-off between entropy and compression, we add training-entropy curves for GRPO and the three LSD variants in Appendix F.1, Figure 21. LSD (SG-FKL) maintains relatively stable training entropy above GRPO even after more than 1,600 training steps.

MethodMean training entropy, steps 1–1628
GRPO0.0641
LSD (SG-FKL)0.0726
LSD (SG-RKL)0.0456
LSD (PG-RKL)0.0576

To examine whether distillation on solved queries restricts subsequent exploration, we add a Countdown combinatorial-search experiment (Appendix B, Figures 15 and 16). This task requires selecting number orderings, operators, and parentheses for multi-step solutions. Each query can admit multiple valid solutions, and correctness can be verified exactly, allowing us to examine success rate, response length, and the structural diversity of correct solutions. We evaluate on 384 fixed test queries, sampling 32 responses per query for each method.

At the final checkpoint, LSD achieves 82.16% overall accuracy, compared with GRPO’s 78.34%, while reducing average response length by 17.28%. On the fixed set of 200 initially hard queries, Pass@1 increases from 61.28% with GRPO to 66.42% with LSD, and Pass@32 increases from 89% to 94%. On the same 326 queries with at least eight correct responses under both methods, we compute the expected number of distinct canonical expression trees among eight correct responses. This value is 2.222 for LSD and 2.100 for GRPO. It shows that LSD improves success on initially hard queries while achieving higher structural diversity.

Countdown metricGRPOLSD
Overall Pass@1, 384 queries (%)78.3482.16
Overall mean response length (tokens)1,446.561,196.64
Response-length reduction vs. GRPO—17.28%
Pass@1 on 200 initially hard queries (%)61.2866.42
Pass@32 on 200 initially hard queries (%)89.0094.00
Expected distinct trees among 8 correct responses, 326 shared queries2.1002.222

Figure 16 provides a concrete example: both methods answer all 32 samples correctly, but LSD produces six correct expression structures compared with GRPO’s three, while reducing average response length by 36.47%.

Figure 16: individual queryGRPOLSD
Correct responses32/3232/32
Distinct canonical expression trees36
Mean response length (tokens)387.75246.34
Response-length reduction vs. GRPO—36.47%
AuthorsOpenReview ↗

Part II of the Authors' Response

W3: Statistical robustness and generalizability remain insufficiently established.

Thank you for the suggestion. We add 20k paired bootstrap resamples at the question level, using identical sampled queries for GRPO and LSD to compute 95% confidence intervals. Appendix G.2, Figure 25 presents results on 22 fixed easy queries, with 32 responses per query per method. Relative to GRPO, SG-RKL and PG-RKL reduce response length by 24.05% and 19.59%, respectively, indicating robust length reductions on this fixed easy-query set.

Method on 22 fixed easy queriesLength reduction vs. GRPO, % [95% CI]Accuracy difference vs. GRPO, percentage points [95% CI]
LSD (SG-RKL)24.05 [18.19, 30.07]+1.28 [+0.43, +2.27]
LSD (PG-RKL)19.59 [13.67, 25.41]+0.99 [−0.14, +2.13]

Regarding your concern about the statistical reliability of the small Pass@1 gains, LSD’s primary contribution is to control response-length growth on solved queries. We do not base our main conclusions on the small Pass@1 gains in the mathematics and agentic tasks. In the additional Countdown experiment (Appendix B, Figures 15 and 16), we evaluate 384 fixed test queries with 32 responses per query. LSD improves overall Pass@1 over GRPO by 3.82 points while reducing average response length by 17.28%. On the same 326 queries with at least eight correct responses under both methods, the expected number of distinct canonical expression trees among eight correct responses is 2.222 for LSD and 2.100 for GRPO. Thus, in this experiment, LSD produces shorter responses while achieving higher success rates and higher structural diversity of correct solutions.

For the LST reference, switching to another stored checkpoint satisfying the accuracy constraint changes each method’s LST by less than 0.091 points.

MethodLST with stored reference step 885 (%)LST with stored reference step 1000 (%)Absolute change (percentage points)
GRPO19.030118.93970.0905
LSD (SG-FKL)−3.7151−3.78820.0732
LSD (SG-RKL)−10.9162−10.98390.0677
LSD (PG-RKL)1.44591.36880.0771

Thanks for the attentive reading of the manuscript and constructive feedback. We have incorporated these changes into our revised version. We hope our response addresses all the concerns and that the reviewer will consider raising the rating accordingly. We are more than glad to answer any further questions.

Dear Reviewer,

As the author-reviewer cycle nears its conclusion, we noticed that we have not yet received your response to our rebuttal. We invested considerable effort in comprehensively addressing each of your concerns, including conducting additional experiments, clarifying technical details and methodological aspects of the paper, and revising the manuscript accordingly.

We would be deeply grateful if you could review our detailed rebuttal and consider whether the substantial improvements and clarifications we have provided address your initial concerns. Your feedback is invaluable to us, and we believe our responses demonstrate significant enhancements to the work's quality and rigor.

Given the approaching deadline, we respectfully request your reconsideration of the manuscript in light of these updates. We sincerely appreciate your time and expertise throughout this process.

Thank you for your review, and we always look forward to your response.

Reviewer_r43TOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper addresses the issue of response-length inflation during reinforcement learning (RL), where models tend to become unnecessarily verbose on already-solved, easy queries without yielding additional performance gains. To measure this side effect, the authors introduce the length-scaling tax (LST), defined as excess response length on solved prompts without a commensurate gain in accuracy. To mitigate this, they propose Length Self-Distillation (LSD), a technique that routes solved prompts to on-policy distillation while retaining the standard RL objective for unsolved prompts. Experimental results demonstrate that LSD effectively maintains conciseness on simpler queries while still enabling efficient exploration on challenging tasks during RL post-training.

Critical strengths
  1. The paper characterizes unnecessary response-length growth on solved prompts through the length-scaling tax (LST). Length Self-Distillation (LSD) applies EMA-teacher distillation to all-correct rollout groups while retaining GRPO for the remaining groups.
  2. investigates factors that amplify LST, helping explain the problem.
  3. validates the efficacy of LSD on both single-turn and multi-turn tasks
Critical weaknesses
  1. The paper does not adequately investigate whether regularizing solved queries via self-distillation depresses the policy’s overall performance ceiling or restricts long-term capability gains across extended training horizons.
  2. The evaluation is conducted strictly on base models. It remains unclear whether the approach generalizes to instruction-tuned models; if an initial checkpoint already exhibits verbose reasoning patterns, distilling against an EMA teacher could inadvertently anchor and preserve that verbosity rather than improve efficiency.
  3. In complex, exploration-intensive domains (e.g., ARC-Challenge or multi-step discovery problems), prematurely anchoring solved trajectories may penalize diverse exploratory search paths, risking policy collapse or suboptimal convergence.
  4. Empirical validation is confined to the Qwen family and small-scale models.
Questions for authors

I would appreciate it if the authors could kindly address the concerns raised in the Critical Weaknesses section.

3 comments
AuthorsOpenReview ↗

Part I of the Authors' Response

Thank you very much for recognizing our characterization of LST, the analysis of factors that amplify it, and LSD’s effectiveness on both single-turn and multi-turn tasks. We appreciate your detailed feedback and address each concern below.

W1: The paper does not adequately investigate whether regularizing solved queries via self-distillation depresses the policy’s overall performance ceiling or restricts long-term capability gains across extended training horizons.

We appreciate your suggestion to examine LSD over longer training horizons. Our main experiments already extend beyond 1,600 training steps. To make the training dynamics clearer, we have added extended-training evaluation curves for GRPO and the three LSD variants on AMC 2023 and AIME 2026 in Section 5.4, Figure 9. At each checkpoint, we sample 32 responses per query. For each method, queries with accuracy ≥0.875 are classified as easy, with the remainder classified as hard.

The three LSD variants continue to improve in aggregate accuracy on these benchmarks during later training. Meanwhile, LSD variants maintain shorter easy-group responses than GRPO at most checkpoints, with an average reduction of 19.86% across the two variants at step 1,600. Their hard-group response lengths continue to increase over training, alongside concise responses on mastered queries.

Method at step 1,600Easy queriesEasy-group length (tokens)Hard-group length (tokens)Easy-length reduction vs. GRPO
GRPO371,230.732,658.62—
LSD (SG-RKL)36919.282,708.1025.31%
LSD (PG-RKL)391,053.362,624.5014.41%

W2 W4: The evaluation is conducted strictly on base models. ... Empirical validation is confined to the Qwen family and small-scale models.

Thank you for encouraging broader evaluation across model families and scales. To examine LSD on instruction-tuned models, a non-Qwen family, and larger parameter scales, we add GRPO-versus-LSD comparisons on Gemma-3-4B-IT and Qwen3-30B-A3B-Instruct-2507. Both comparisons follow the core training settings of the main experiments and use a 4K maximum response length. Results are reported in Appendix A.

  • On Gemma, GRPO post-training still exhibits LST: response length increases without sustained accuracy gains. LSD controls length growth more effectively early in training and achieves aggregate accuracy 2.91 points higher than GRPO (Appendix A.1, Figure 11). We further analyze statistics over all evaluation responses and the distributions of 32 responses per query (Figures 12 and 13). We find that GRPO-trained Gemma repeatedly rewrites endings after producing correct answers and continues ineffective self-checking after incorrect problem formulation. LSD substantially alleviates these behaviors.

  • On Qwen3-30B-A3B, GRPO itself improves accuracy while shortening responses under the 4K cap, including on the fixed initially easy queries (Appendix A.3, Figure 14). This setting does not exhibit the LST phenomenon that motivates LSD. We have clarified in the Limitations paragraph of Section 6 that LSD targets LST emerging during training, so its applicability should be assessed based on the observed training dynamics.

ModelStepMethodGPUsTraining time (h)Overall Pass@1 (%)Mean response length (tokens)
Gemma-3-4B-IT450GRPO8 × H2017.8029.4582,766.19
Gemma-3-4B-IT450LSD8 × H2018.6632.3652,253.09
Qwen3-30B-A3B300GRPO64 × H2013.1563.1342,432.42
Qwen3-30B-A3B300LSD64 × H2014.9760.6642,575.82
AuthorsOpenReview ↗

Part II of the Authors' Response

W3: In complex, exploration-intensive domains, prematurely anchoring solved trajectories may penalize diverse exploratory search paths, risking policy collapse or suboptimal convergence.

We appreciate the opportunity to address this concern about exploration. To examine whether distillation on solved queries restricts further exploration, we add a controlled Countdown combinatorial search experiment (Appendix B, Figures 15 and 16). The task requires multi-step choices of number ordering, operators, and parentheses, can admit multiple valid solutions to the same problem, and permits exact verification. This allows us to jointly evaluate success rates, response length, and the structural diversity of correct solutions. Both methods use Qwen3-4B-Base with the same training budget and are evaluated on 384 fixed held-out queries with 32 responses per query. LSD routes a group to distillation only when all current rollouts are correct; the remaining groups continue to use GRPO.

At the final evaluation, LSD achieves 82.16% overall accuracy versus GRPO’s 78.34%, while reducing mean response length by 17.28%. On the same 200 initially hard queries, Pass@1 improves from 61.28% under GRPO to 66.42% under LSD, and Pass@32 from 89% to 94%. Interestingly, on the same 326 queries where both methods obtain at least eight correct responses, we compute the expected number of distinct canonical expression trees among eight correct responses: 2.222 for LSD versus 2.1 for GRPO. These results show that LSD improves success on initially hard queries while also achieving slightly higher structural diversity among its final correct solutions.

Countdown metricGRPOLSD
Overall Pass@1, 384 queries (%)78.3482.16
Overall mean response length (tokens)1,446.561,196.64
Response-length reduction vs. GRPO—17.28%
Pass@1 on 200 initially hard queries (%)61.2866.42
Pass@32 on 200 initially hard queries (%)89.0094.00
Expected distinct trees among 8 correct responses, 326 shared queries2.1002.222

Figure 16 provides a concrete example: both methods answer all 32 samples correctly, but LSD produces six correct expression structures compared with GRPO’s three, while reducing average response length by 36.47%.

Figure 16: individual queryGRPOLSD
Correct responses32/3232/32
Distinct canonical expression trees36
Mean response length (tokens)387.75246.34
Response-length reduction vs. GRPO—36.47%

Thanks for the attentive reading of the manuscript and constructive feedback. We have incorporated these changes into our revised version. We hope our response addresses all the concerns and that the reviewer will consider raising the rating accordingly. We are more than glad to answer any further questions.

Dear Reviewer,

As the author-reviewer cycle nears its conclusion, we noticed that we have not yet received your response to our rebuttal. We invested considerable effort in comprehensively addressing each of your concerns, including conducting additional experiments, clarifying technical details and methodological aspects of the paper, and revising the manuscript accordingly.

We would be deeply grateful if you could review our detailed rebuttal and consider whether the substantial improvements and clarifications we have provided address your initial concerns. Your feedback is invaluable to us, and we believe our responses demonstrate significant enhancements to the work's quality and rigor.

Given the approaching deadline, we respectfully request your reconsideration of the manuscript in light of these updates. We sincerely appreciate your time and expertise throughout this process.

Thank you for your review, and we always look forward to your response.

Reviewer_jPgFOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies a specific issue in RL post-training: when the model learns longer reasoning trajectories on hard problems, its responses to easy problems that it can already solve also tend to grow longer, without commensurate accuracy gains. The authors propose the Length-Scaling Tax (LST), which quantifies this excess length using a held-out set of easy problems and a reference checkpoint with satisfactory accuracy.To mitigate this effect, the authors introduce Length Self-Distillation (LSD): groups of fully correct rollouts from the current iteration are routed to on-policy distillation with an EMA teacher, while the remaining groups continue training with the original RL objective. Experiments cover single-turn mathematical reasoning and multi-turn search tasks, with ablations over three distillation objectives, teacher half-lives, and routing thresholds.

Critical strengths
  1. Tracking performance separately on already-solved problems reveals behavioral degradation masked by aggregate average length metrics. Using a fixed evaluation set avoids confounding comparisons caused by dynamically reselecting easy problems as model capability improves.
  2. The method is well-motivated. Fully correct rollout groups receive zero relative advantage and thus lack reward gradients that directly regularize their behavior. Adding token-level supervision on these groups is a sensible design. LSD requires no external strong teacher and does not rely on explicit length rewards. Paired evaluations support the core phenomenon, and the paper includes illustrative failure cases for individual problems, leading to a balanced presentation of results.
Critical weaknesses
  1. The fixed-teacher ablation alters routing at the same time, making it impossible to isolate the effect of the EMA teacher. Online routing may assign problems newly mastered by the student but not yet reliably solved by the initial teacher to the distillation branch. Since Fixed SG-FKL uses both a frozen teacher and a static routing mapping, the gap between Fixed SG-FKL and LSD may stem either from teacher updates or from routing differences.
  2. The multi-turn experiments report only one operating point for RL + Length Penalty: it achieves lower LST but substantially lower accuracy. This demonstrates aggressive compression at that setting and does not establish that length penalty remains inferior at matched accuracy or comparable length relative to LSD. In addition, the extra forward passes, memory footprint, and training overhead introduced by the EMA teacher are not quantified. The LST metric for the multi-turn task excludes tool observations and refinements generated by the model, so it does not directly reflect end-to-end computational cost.
Questions for authors
  1. With online routing, loss weights, and training budget held identical, what is the performance gap between a frozen initial teacher and an EMA teacher? Does the result still support the current explanation for the failure of the Fixed baseline?
  2. How were checkpoints selected for Table 2? What training step counts and selection rules correspond to the 44.20% of SG-FKL in the main experiment versus the 41.85% from the repeated run under the same configuration in Table 4?
  3. Are the conclusions robust after paired confidence interval estimation for the main length improvements and accuracy gaps across problems? Does the LST value change materially when the reference checkpoint is re-sampled independently?
  4. When the length penalty baseline is tuned to reach accuracy or compression levels comparable to LSD, does LSD retain its advantage? What is the cost–benefit profile once the overhead of the teacher and the multi-turn system is accounted for?
3 comments
AuthorsOpenReview ↗

Part I of the Authors’ Response

Thank you very much for recognizing our analysis of already-solved problems and the motivation behind LSD. We appreciate your detailed feedback and address your concerns below through additional experiments, uncertainty analyses, and clarifications.

W1 Q1: With online routing, loss weights, and training budget held identical, what is the performance gap between a frozen initial teacher and an EMA teacher? Does the result still support the current explanation for the failure of the Fixed baseline?

We appreciate your suggestion to separate the roles of teacher updates and online routing. To examine this, we add an SG-FKL ablation with a frozen teacher and online routing. As in LSD, a query is routed to the OPD loss only when all responses in its current rollout group are correct. The results are reported in Appendix E.3, Figure 20.

At 600 training steps, the frozen teacher with online routing maintains short responses (1,304 tokens on average), but its accuracy remains below that of EMA-based LSD: 29.26% versus 37.22%. As Figure 4 shows, queries considered easy by the updated policy may not be reliably solved by the initial teacher. The frozen teacher may therefore provide outdated distillation targets for queries selected by online routing. The EMA teacher evolves with the student to help reduce this capability mismatch.

Teacher with SG-FKL and online routingStepOverall Pass@1 (%)Mean response length (tokens)
Frozen initial teacher60029.261,303.77
EMA teacher (LSD)60037.222,053.80

W2 Q4: When the length penalty baseline is tuned to reach accuracy or compression levels comparable to LSD, does LSD retain its advantage? What is the cost–benefit profile once the overhead of the teacher and the multi-turn system is accounted for?

Thank you for drawing attention to both the accuracy and length trade-off and the additional computational cost. We agree that a single multi-turn RL + length-penalty operating point does not establish LSD’s superiority at matched accuracy or overall length. We therefore add a supplementary single-turn LCPO comparison with three penalty strengths (α = 3×10⁻⁵, 10⁻⁴, and 3×10⁻⁴; Section 5.5, Figure 10). At 600 steps, their accuracies are 30.33%, 32.36%, and 30.44%, respectively, below LSD’s 37.22%, despite shorter outputs. Within the tested range, weakening the penalty does not eliminate this accuracy gap.

Method at step 600Penalty strengthOverall Pass@1 (%)Mean response length (tokens)
LCPOα = 3×10⁻⁵30.33887.76
LCPOα = 10⁻⁴32.361,127.34
LCPOα = 3×10⁻⁴30.44788.76
LSD (SG-FKL)—37.222,053.80

The added training overhead is modest in our measured setting. LSD reuses student rollouts and adds a teacher forward pass without gradients and an EMA parameter update. It requires no separate teacher generation, backward pass, or optimizer. On Countdown, using Qwen3-4B-Base and eight H20 GPUs per method, the average training-cycle time over the first 500 updates increases by 8.46%, measured using each method’s actual rollouts. Additional sharded FP32 teacher parameter storage is approximately 1.87 GiB per GPU. The teacher is used only during training and adds no computation at inference (Appendix E.4, Table 6).

QuantityGRPOLSD (SG-FKL)
Training cycle (s/update)103.22111.95
Training-cycle overhead vs. GRPO—8.46%
Teacher forward pass (s/update)—6.19
Extra FP32 teacher parameter storage (GiB/GPU)01.87

Q2: How were checkpoints selected for Table 2? What training step counts and selection rules correspond to the 44.20% of SG-FKL in the main experiment versus the 41.85% from the repeated run under the same configuration in Table 4?

We welcome the opportunity to clarify the checkpoint selection rules. For each method in Table 2, we select one checkpoint with the highest overall average Pass@1 during training. The 44.20% reported for SG-FKL comes from this best checkpoint. Table 4 instead reports all ablation results at step 1750, allowing configurations to be compared at the same training step. An independent run with the same SG-FKL configuration achieves 41.85% at this step. The two tables therefore use different checkpoint selection rules. The fixed step-1750 setting is stated in the manuscript and the caption of Table 4.

AuthorsOpenReview ↗

Part II of the Authors’ Response

Q3: Are the conclusions robust after paired confidence interval estimation for the main length improvements and accuracy gaps across problems? Does the LST value change materially when the reference checkpoint is re-sampled independently?

We agree that uncertainty estimates are needed to assess the reliability of these comparisons. We use questions as the resampling unit and apply identical sampled query indices to GRPO and LSD, computing 95% confidence intervals from 20k paired bootstrap resamples. Appendix G.2, Figure 25 presents results on 22 fixed easy queries, with 32 rollouts per query per method. Relative to GRPO, LSD (SG-RKL) and LSD (PG-RKL) reduce response length by 24.05% and 19.59%, respectively. Their accuracy differences are +1.28 points and +0.99 points.

Method on 22 fixed easy queriesLength reduction vs. GRPO, % [95% CI]Accuracy difference vs. GRPO, percentage points [95% CI]
LSD (SG-RKL)24.05 [18.19, 30.07]+1.28 [+0.43, +2.27]
LSD (PG-RKL)19.59 [13.67, 25.41]+0.99 [−0.14, +2.13]

For the reference checkpoint, we also substitute another stored checkpoint that satisfies the accuracy constraint and find that each method’s LST changes by less than 0.091 points.

MethodLST with stored reference step 885 (%)LST with stored reference step 1000 (%)Absolute change (percentage points)
GRPO19.030118.93970.0905
LSD (SG-FKL)−3.7151−3.78820.0732
LSD (SG-RKL)−10.9162−10.98390.0677
LSD (PG-RKL)1.44591.36880.0771

Thanks for the attentive reading of the manuscript and constructive feedback. We have incorporated these changes into our revised version. We hope our response addresses all the concerns and that the reviewer will consider raising the rating accordingly. We are more than glad to answer any further questions.

Dear Reviewer,

As the author-reviewer cycle nears its conclusion, we noticed that we have not yet received your response to our rebuttal. We invested considerable effort in comprehensively addressing each of your concerns, including conducting additional experiments, clarifying technical details and methodological aspects of the paper, and revising the manuscript accordingly.

We would be deeply grateful if you could review our detailed rebuttal and consider whether the substantial improvements and clarifications we have provided address your initial concerns. Your feedback is invaluable to us, and we believe our responses demonstrate significant enhancements to the work's quality and rigor.

Given the approaching deadline, we respectfully request your reconsideration of the manuscript in light of these updates. We sincerely appreciate your time and expertise throughout this process.

Thank you for your review, and we always look forward to your response.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Overview

Dear PC, SAC, AC, and all reviewers,

We sincerely thank you for your thoughtful reviews and constructive feedback. We appreciate the reviewers' recognition of our controlled analysis of LST, the motivation behind LSD, and its empirical support across reasoning and agentic tasks.

We have prepared point-by-point responses and expanded the manuscript with the following experiments and analyses. Revised text is marked in blue.

AreaAdditions and revisionsLocationReviewer IDs
Broader model coverageAdded GRPO-versus-LSD comparisons on Gemma-3-4B-IT and Qwen3-30B-A3B-Instruct-2507, extending evaluation to instruction-tuned models, a non-Qwen family, and a larger parameter scale. Added response statistics and individual case analyses.Appendix A, Figures 11–14r43T, F19W
Continued learningAdded evaluation curves beyond 1,600 training steps, comparing GRPO and three LSD variants on accuracy and response lengths for easy and hard queries.Section 5.4, Figure 9r43T, F19W
Exploration and solution diversityAdded controlled Countdown experiments on 384 queries × 32 responses. LSD improves Pass@1 by 3.82 percentage points with 17.28% shorter responses and slightly higher correct-solution structural diversity. Added an illustrative expression-tree comparison.Appendix B, Figures 15–16r43T, F19W
Stronger efficiency baselinesAdded DAST and LCPO, each evaluated at three length-control strengths, together with an analysis of DAST’s adaptation to a base model.Appendix C, Figure 17; Section 5.5, Figure 10jPgF, F19W
Teacher-update ablationAdded a frozen teacher with online routing comparison against EMA-based LSD. Updated the main comparison to include Fixed LSD with both static and online routing.Appendix E.3, Figure 20; Figure 6jPgF
Entropy analysisAdded GRPO to the training-entropy comparison with all three LSD variants over steps 1–1628. SG-FKL retains relatively stable training entropy above GRPO late in training.Appendix F.1, Figure 21F19W
Training overheadAdded a measured Countdown cost comparison: 8.46% additional training-cycle time and approximately 1.87 GiB of sharded FP32 teacher parameter storage per GPU. Clarified that the teacher is used only during training.Appendix E.4, Table 6jPgF
Statistical uncertaintyAdded 20,000 question-level paired bootstrap resamples on 22 fixed easy queries, with 32 responses per query per method. Reported confidence intervals for length reduction and accuracy differences alongside individual rollout lengths.Appendix G.2, Figure 25jPgF, F19W
Reference-checkpoint sensitivityExamined substitution with another accuracy-qualified reference checkpoint. Each method’s LST changes by less than 0.091 percentage points.Reviewer responsesjPgF, F19W
Reporting clarity and applicabilityClarified best-checkpoint selection in Table 2 versus fixed-step evaluation in Table 4, the scope of LST as a policy-output metric, and LSD’s applicability to training settings where LST emerges.Tables 2 and 4; Appendix E.4; Section 6jPgF, r43T

Thank you again for your time and consideration. We welcome further discussion of these additions and clarifications.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.