- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies whether length-controlled RL can reduce inference tokens while preserving aggregate accuracy yet induce regressions on individual evaluation items. Starting from shared Qwen3.5 checkpoints, the authors compare mild and strong length penalties, test whether initially degraded items recover under fixed-weight inference interventions, and then continue training with the strong penalty kept, relaxed, or removed. The paper further evaluates whether taskwise regression information changes the selected training continuation. The main finding is that strong length pressure can substantially reduce average token use at similar aggregate accuracy while some individual items regress; some regressions persist under the tested inference controls but improve after relaxing or removing the penalty.
Critical strengths
- The experimental protocol is careful and unusually extensive. The study tracks the same items across paired training branches, tests fixed-weight inference interventions before modifying training, includes matched panel-level inference-cost comparisons, and evaluates continuation choices on held-out tasks.
- The evaluation includes multiple training seeds, model scales, domains, and a large repeated-sampling budget, making the empirical study substantially more systematic than a standard aggregate accuracy/token comparison.
Critical weaknesses
- The main scientific conclusions appear limited in novelty. A strong training-time length penalty changes the learned policy, so it is not surprising that simply allocating more inference tokens does not necessarily undo all resulting regressions. Likewise, after cloning the strong-penalty checkpoint, continuing RL for 400 updates with a weaker or removed regularizer can naturally improve some previously degraded examples. The paper measures these effects carefully, but does not introduce a new training method or provide a mechanism that substantially changes our understanding of length-controlled RL.
- The “taskwise” analysis is item-level rather than explanatory. The paper identifies individual task IDs whose estimated success probability decreases, but does not characterize which kinds of problems are systematically vulnerable or why—for example by difficulty, baseline reasoning length, reasoning structure, or task category. Consequently, the results provide limited predictive guidance beyond monitoring each evaluation item separately. Relatedly, the reported “recovery” after another 400 RL updates does not distinguish restoration of a previously damaged capability from ordinary continued learning or relearning. The matched-cost experiment controls only panel-average token use, while token allocation may still differ across the recovered items themselves.
- Some reported statistics are unusually regular and would benefit from further clarification. In Table 5, the S–M accuracy differences stay extremely close to +0.20 pp across both domains and all three model scales, with very small across-seed SDs. In Table 6, the reported accuracy SDs across many distinct benchmark/checkpoint conditions repeatedly take values only around 0.5–0.7 despite substantially different datasets and sample sizes. These patterns are not impossible given the large repeated-sampling budget, but unrounded per-seed values would help establish confidence in the reported summaries.
Questions for authors
- Can the authors characterize which types of tasks are vulnerable to strong length pressure, rather than only which individual task IDs regress? For example, do regressions correlate with baseline reasoning length, difficulty, puzzle size/structure, or other interpretable task properties, and are the same vulnerable items consistent across training seeds?
- How should the continuation results be distinguished from ordinary additional RL or relearning? In particular, is there evidence that Relax/Remove restore a previously lost capability rather than simply learning those examples again under a more permissive objective? For the matched-cost analysis, are inference tokens also comparable on the recovered items themselves, rather than only in panel average?
- Could the authors provide the unrounded per-seed measurements underlying Tables 1, 5, and 6, particularly for the near-identical S–M accuracy gaps and the highly regular cross-seed SDs?