acceptodds
Under review as a conference paper at ICLR 2027

Token Savings Can Mask Recoverable Taskwise Regressions in Length-Controlled RL

Abstract

Length-controlled reinforcement learning (RL) aims to learn token-efficient chains of thought while preserving task accuracy. However, average length and aggregate accuracy can conceal taskwise regressions, leaving practitioners uncertain whether to allocate more inference tokens or modify training. We introduce a taskwise intervention framework that connects regression diagnosis to training decisions. It tracks the same tasks through fixed-weight inference interventions and training continuations that retain, relax, or remove length penalties, comparing recovery at matched inference cost. Across the Qwen3.5-4B logic and code experiments, strong fixed soft-overlong penalties on all responses achieve at least 30.82% token savings at equivalent aggregate accuracy, yet at least 11.0% of tasks remain degraded across the tested inference controls. In each training seed, a single modified-penalty branch recovers at least a 6.0% share of the full task panel from these persistent deficits, outperforming continued strong-penalty training at matched inference cost. Across five seeds per domain, taskwise evidence selects Relax over Keep in all ten runs, reducing prespecified held-out decision loss by 11.30% on logic and 13.77% on code on average, with 15.2% and 18.0% more inference tokens. These gains include minimum recorded monitoring and exploratory-training costs; token-heavy weights reverse this preference. The framework connects taskwise regression and recovery to continuation choices beyond aggregate token savings.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Model Reasoning, Length-Controlled Reinforcement Learning, Token Complexity, Identifiability, Token Budget Allocation, Capability Evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_H5jhOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether length-controlled RL can reduce inference tokens while preserving aggregate accuracy yet induce regressions on individual evaluation items. Starting from shared Qwen3.5 checkpoints, the authors compare mild and strong length penalties, test whether initially degraded items recover under fixed-weight inference interventions, and then continue training with the strong penalty kept, relaxed, or removed. The paper further evaluates whether taskwise regression information changes the selected training continuation. The main finding is that strong length pressure can substantially reduce average token use at similar aggregate accuracy while some individual items regress; some regressions persist under the tested inference controls but improve after relaxing or removing the penalty.

Critical strengths
  1. The experimental protocol is careful and unusually extensive. The study tracks the same items across paired training branches, tests fixed-weight inference interventions before modifying training, includes matched panel-level inference-cost comparisons, and evaluates continuation choices on held-out tasks.
  2. The evaluation includes multiple training seeds, model scales, domains, and a large repeated-sampling budget, making the empirical study substantially more systematic than a standard aggregate accuracy/token comparison.
Critical weaknesses
  1. The main scientific conclusions appear limited in novelty. A strong training-time length penalty changes the learned policy, so it is not surprising that simply allocating more inference tokens does not necessarily undo all resulting regressions. Likewise, after cloning the strong-penalty checkpoint, continuing RL for 400 updates with a weaker or removed regularizer can naturally improve some previously degraded examples. The paper measures these effects carefully, but does not introduce a new training method or provide a mechanism that substantially changes our understanding of length-controlled RL.
  2. The “taskwise” analysis is item-level rather than explanatory. The paper identifies individual task IDs whose estimated success probability decreases, but does not characterize which kinds of problems are systematically vulnerable or why—for example by difficulty, baseline reasoning length, reasoning structure, or task category. Consequently, the results provide limited predictive guidance beyond monitoring each evaluation item separately. Relatedly, the reported “recovery” after another 400 RL updates does not distinguish restoration of a previously damaged capability from ordinary continued learning or relearning. The matched-cost experiment controls only panel-average token use, while token allocation may still differ across the recovered items themselves.
  3. Some reported statistics are unusually regular and would benefit from further clarification. In Table 5, the S–M accuracy differences stay extremely close to +0.20 pp across both domains and all three model scales, with very small across-seed SDs. In Table 6, the reported accuracy SDs across many distinct benchmark/checkpoint conditions repeatedly take values only around 0.5–0.7 despite substantially different datasets and sample sizes. These patterns are not impossible given the large repeated-sampling budget, but unrounded per-seed values would help establish confidence in the reported summaries.
Questions for authors
  1. Can the authors characterize which types of tasks are vulnerable to strong length pressure, rather than only which individual task IDs regress? For example, do regressions correlate with baseline reasoning length, difficulty, puzzle size/structure, or other interpretable task properties, and are the same vulnerable items consistent across training seeds?
  2. How should the continuation results be distinguished from ordinary additional RL or relearning? In particular, is there evidence that Relax/Remove restore a previously lost capability rather than simply learning those examples again under a more permissive objective? For the matched-cost analysis, are inference tokens also comparable on the recovered items themselves, rather than only in panel average?
  3. Could the authors provide the unrounded per-seed measurements underlying Tables 1, 5, and 6, particularly for the near-identical S–M accuracy gaps and the highly regular cross-seed SDs?
Reviewer_L3UMOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper introduces a taskwise framework for length-controlled RL that detects regressions hidden by aggregate accuracy and token savings, compares inference interventions with training changes, and guides whether to retain, relax, or remove length penalties. Evaluation seems extensive across multiple seeds.

Critical strengths
  1. Consistent evidence across multiple seeds. The main experiments use five training seeds in both logic and code, and increase appears to be stably improved.

  2. An interesting and practically relevant question. The paper investigates whether token savings conceal regressions on individual tasks despite similar aggregate accuracy, and whether inference interventions or training changes can recover them.

Critical weaknesses
  1. Poor readability. The main text is overloaded with procedural details, such as SHA-256 task-ID sorting, sampling ledgers, and numerous thresholds. Yet the central claim and contributions are not clearly emphasized and validated. This would make the work appear more like an implementation report rather than scientific paper.

  2. From my personal understanding, the performance gain compared to the selected hyperparameter appears marginal. Table 5 reports a joint-recovery lower bound of 11% under this recipe and only 1% with progressive pressure restricted to correct responses.

  3. Regarding to the novelty aspect, this submission does not adequately position its taskwise evaluation against JET [1] or establish what its diagnosis-and-recovery framework adds beyond existing evaluation and penalty-control strategies.

[1] Han, Jinyi, Ying Huang, Ying Liao, et al. “Your Models Have Thought Enough: Training Large Reasoning Models to Stop Overthinking.” In International Conference on Learning Representations, 2026.

Questions for authors

I believe the paper would benefit considerably from a revision focused on readability. It would be important to have a human reader in mind during writing, instead of AI.

Reviewer_GENqOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates whether token savings and preserved aggregate accuracy in length-controlled reinforcement learning can conceal regressions on individual tasks. The authors compare mild and strong length penalties, identify degraded tasks, test fixed-weight inference interventions, and continue the strong-penalty checkpoint under Keep, Relax, and Remove branches. They also compare aggregate-based checkpoint selection with a selector incorporating taskwise risk. Experiments primarily use Qwen3.5-4B on logic and code, with additional model-scale and benchmark evaluations.

Critical strengths

S1. A practically relevant evaluation concern. The paper highlights a limitation of assessing length-controlled RL solely through aggregate accuracy and token savings. Tracking success probabilities on individual tasks provides a more detailed picture of which tasks are preserved or degraded.

S2. Well-controlled recovery experiments. Keep, Relax, and Remove start from the same training state and receive the same number of additional updates. The comparison against Keep helps isolate the effect of changing the length penalty, while matched average inference costs strengthen the evidence that recovery is not solely due to longer outputs.

S3. Careful treatment of uncertainty and selection. The paper uses simultaneous confidence bounds to assess taskwise changes and evaluates validation-selected continuations on independent test tasks. It also reports monitoring and exploratory-training costs and examines sensitivity to cost weights and deployment horizons, making the trade-offs more explicit.

Critical weaknesses

W1. Limited methodological novelty. The framework combines DAPO-style soft-overlong penalties, prompt-specified token limits, s1-style budget forcing, and continued training with a different penalty coefficient λ. Direct adds a taskwise-risk threshold to Aggregate’s accuracy requirement. While this combination provides a useful evaluation protocol, the paper does not clearly establish a new training or inference algorithm.

W2. The main findings offer limited guidance beyond confirming plausible outcomes. For C1, the paper’s own example of (0.9, 0.5) versus (0.5, 0.9) illustrates why equal average accuracy can conceal taskwise regressions. The experiments quantify this effect, but do not establish when substantial regressions should be expected. For C2, relaxing or removing an overly strong length penalty is a natural intervention for recovering performance. The Keep comparison and matched-cost evaluation make this finding more convincing, but the results do not yield a transferable rule for when to relax the penalty or how to choose its strength.

W3. The core evidence comes from a narrow training setup. The main experiments use one model family, two training domains, one mild/strong penalty pair, one branching point (t* = 600), and one continuation length (H = 400). Table 5 provides useful extensions, but Table 5b also reveals substantial sensitivity to the penalty recipe: combining correctness gating with a progressive schedule reduces the joint recovery lower bound from 11% to 1%. This does not eliminate taskwise regression, but suggests that the headline recovery results may depend strongly on fixed penalties applied to all responses. Experiments across model families, penalty strengths, and branching schedules are needed to clarify the generality of the findings.

W4. Taskwise diagnosis stops at identifying and counting regressions. The paper provides little characterization of initially degraded or persistently degraded tasks. It would be informative to examine task difficulty—for example, the number of houses/categories in ZebraLogic and difficulty levels in LiveCodeBench—reference-model output length, and failure types such as truncation or answer-format errors. Comparisons of reasoning trajectories could further test whether regression involves omitted verification, premature termination, or changes in solution strategy. Such analyses would help explain what the penalty changes in model behavior, rather than only whether performance can subsequently be restored.

Questions for authors

Q1. Clarification on dependence on the penalty recipe. Table 5b shows that when correctness gating and progressive scheduling are combined, the Loss LB falls from 22% to 9% and the Joint LB falls from 11% to 1%. How much of the remaining drop comes from fewer persistent regressions under inference controls, and how much from less recovery when the penalty is changed? This would help assess whether the headline finding extends beyond fixed penalties applied to all responses.

Q2. Clarification on the scope of inference persistence. Among tasks classified as persistently degraded, how often did the tested controllers actually increase output length, and by how much? If they often failed to elicit additional tokens, the result may reflect the limited effectiveness of the controllers rather than a failure to recover despite additional inference computation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.