acceptodds
Under review as a conference paper at ICLR 2027

Quality Determines Direction, Length Shapes Magnitude: Length Control for Open-Ended Reinforcement Learning

Abstract

Reinforcement learning (RL) changes not only what language models say, but also how much they say, often increasing response length at the cost of token efficiency. Controlling this length growth is particularly challenging in open-ended RL because **(i)** response length is entangled with quality, **(ii)** open-ended tasks lack a natural success boundary for deciding when efficiency should be prioritized, and **(iii)** dense, graded rewards often yield small within-group quality margins, making quality-induced advantages especially sensitive to reward-level length shaping, which can perturb their magnitudes and even reverse their signs. We therefore adopt an asymmetric principle: **quality should determine the direction of reinforcement, while length should only shape its magnitude**. We instantiate this principle with **Quality-Gated Length Advantage Shaping (QGLAS)**, which first computes advantages from quality rewards alone, then adds bounded bonuses only to shorter positive-advantage responses, leaving all other advantages unchanged. The bonus strength is further adapted to within-group quality separation, allowing conciseness to matter more when quality-favored responses are similar and less when their quality differences are clear. Across different model families, open-ended benchmarks, and reward sources, QGLAS consistently achieves a stronger quality–length trade-off than representative baselines. At approximately 30% compression, QGLAS retains 98.4–102.0% of the macro-average quality gains achieved by quality-only RL over the base model, compared with 68.3–75.5% for these baselines at comparable compression.

Primary area:
reinforcement learning
Keywords:
Large Language Models, Reinforcement Learning, LLM Post-Training,Length Control, Reward Shaping

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ifGDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies response-length control in open-ended LLM reinforcement learning, arguing that under dense, graded quality rewards, adding length bonuses at the reward level can flip the sign of group-relative advantages for responses near the group mean. The authors propose Quality-Gated Length Advantage Shaping (QGLAS, Section 3), which modifies advantages directly via A~i=Aiq+λghi\widetilde{A} _ i = A _ i^q + \lambda _ g h _ i (Eq. 1): it restricts length bonuses to the quality-favored subset P={i:Aiq>0}\mathcal{P} = \lbrace i : A _ i^q > 0 \rbrace (positive-only gating), assigns bounded one-sided bonuses only to responses shorter than the positive-set average length LrefL _ {\mathrm{ref}} (Eq. 4), and scales the group bonus λg\lambda _ g using the overall reward spread sall=Q100(r)−Q25(r)s _ {\mathrm{all}} = Q _ {100}(r) - Q _ {25}(r) and the positive-set spread s+s _ + (Eqs. 6–9) so that sign(A~i)=sign(Aiq)\mathrm{sign}(\widetilde{A} _ i) = \mathrm{sign}(A _ i^q) (Eq. 5).

QGLAS is tested under 1,000-step GSPO training on 13K open-ended prompts (Skywork-Reward-V2-Llama-3.1-8B) with Qwen3-4B and GLM-4.7-Flash (30B-A3B), evaluated on IFBench, Arena-Hard-v2 Hard Prompts, and Arena-Hard-v2 Creative Writing (Section 4, Table 1). While the paper reports headline macro-average Quality-Gain Retention (QGR, Eq. 10) of 102.0% (Qwen3-4B) and 98.4% (GLM-4.7-Flash) at roughly 32% length compression—alongside offline sign-flip diagnostics (Figure 3a) and ablations (Table 2)—a closer inspection of Table 1 reveals that this "near-lossless" macro QGR is heavily distorted by benchmark scale imbalance (Creative Writing accounts for over 91% of the net denominator on Qwen3-4B, while both IFBench and Hard Prompts actually degrade relative to NoBonus), alongside a mismatch between the standardized advantage theory in Section 3.1/3.3 and the unnormalized GSPO implementation and single-seed reporting for GLM-4.7-Flash and alternative rewards.

Critical strengths

1. Clear diagnosis of advantage sign reversals under dense vs. binary rewards.
Section 4.3 (Figure 3a–b) identifies a concrete reason why reward-level length mixing (GR3\mathrm{GR}^3, GRLC) degrades open-ended RL more severely than binary RLVR: dense rewards cluster near the group mean rˉ\bar{r}, causing the group-mean shift −δˉ-\bar{\delta} from length bonuses to flip the sign of 5%–19%5\%\text{--}19\% of rollout advantages, and manually overriding just those flipped signs during GR3\mathrm{GR}^3 training recovers a large fraction of the lost score (75.50%→86.82%75.50\% \to 86.82\% QGR under Zero in Figure 3b).

2. Clean structural ablations and consistent 3-seed gains over GR3\mathrm{GR}^3 and GRLC on Qwen3-4B.
Within the Qwen3-4B learned-reward setting, the matched-compression comparison against GR3\mathrm{GR}^3 and GRLC is backed by 3 training seeds (Appendix Table 6: macro score 32.28±0.5732.28 \pm 0.57 vs. 29.20±0.5129.20 \pm 0.51 and 29.14±0.5329.14 \pm 0.53), and Table 2 (along with compression-matched Appendix Table 8) systematically ablates advantage-level shaping, positive-only gating, one-sided shaping, and adaptive scaling.

Critical weaknesses

1. The headline "near-lossless (98.4%–102.0%98.4\%\text{--}102.0\%) quality retention" claim is an artifact of unweighted macro-averaging dominated by Creative Writing, while objective instruction following (IFBench) and Hard Prompts both degrade.
As noted under Soundness, the central empirical claim of the paper relies on the aggregate QGR metric in Eq. (10). However, inspecting the individual benchmark columns in Table 1 tells a very different story:

  • On Qwen3-4B, quality-only RL (NoBonus) already degrades strict instruction following on IFBench by −2.33-2.33 points relative to Base (29.22→26.8929.22 \to 26.89), and QGLAS degrades it further to 25.9625.96 (−0.93-0.93 vs. NoBonus, −3.26-3.26 vs. Base). On Hard Prompts, QGLAS also drops relative to NoBonus (20.4020.40 vs. 21.0321.03, retaining 88.3%88.3\% of the +5.40+5.40 gain). The only reason QGLAS's macro QGR exceeds 100%100\% (102.0%102.0\%) is that Creative Writing leaps by +31.67+31.67 points from Base (16.5316.53) to NoBonus (48.2048.20)—accounting for 91.2%91.2\% of the entire macro QGR denominator—and QGLAS scores 50.4750.47 (+2.27+2.27) on that single GPT-4.1-judged benchmark.
  • On GLM-4.7-Flash, the exact same pattern occurs: IFBench plummets by −12.89-12.89 points from Base (53.0053.00) to NoBonus (40.1140.11) and drops another −1.11-1.11 points under QGLAS (39.0039.00), while Hard Prompts also falls from 60.2360.23 (NoBonus) to 59.3059.30 (QGLAS, −0.93-0.93).
  • Even in Table 3 (Rubric-based Judge on Qwen3-4B), IFBench drops from 29.2229.22 (Base) to 28.5928.59 (NoBonus) and 26.8026.80 (QGLAS, −1.79-1.79 vs. NoBonus), while Hard Prompts drops from 25.3025.30 (NoBonus) to 23.1023.10 (QGLAS, −2.20-2.20), yet the paper still reports a high macro QGR (93.3%93.3\%) because Creative Writing rose from 16.5316.53 to 55.3355.33. Because an unweighted average where one subjective creative-writing benchmark carries >90%>90\% of the denominator weight masks consistent regressions on IFBench and Hard Prompts across every model and reward source, the current metric and framing significantly overstate across-the-board quality preservation. This directly drives my weak-reject rating.

2. Missing statistical uncertainty for GLM-4.7-Flash, alternative reward sources (Table 3), and causal interventions (Figure 3b), plus incomplete comparison to simple sign-corrected baselines.
While the Qwen3-4B learned-RM main row uses 3 seeds, GLM-4.7-Flash (Table 1, trained once on seed 42 and merely re-evaluated 3 times, Appendix A.1 lines 681–683), the Rubric and LLM-as-a-Judge experiments in Table 3, the targeted sign-reversal fixes in Figure 3b, and the Pareto sweep points in Figure 1 / Appendix Table 7 are all single-seed runs without prompt-level bootstrap confidence intervals. Furthermore, Figure 3b shows that simply zeroing out sign-flipped advantages in GR3\mathrm{GR}^3 (GR3 + Zero) already jumps QGR from 75.50%75.50\% to 86.82%86.82\% at the same 32.53%32.53\% compression, and in Table 2, QGLAS w/o Advantage-Level Shaping (which keeps positive-only one-sided bonuses at the reward level) reaches 87.9%87.9\% QGR at 32.5%32.5\% CR with only a 0.99%0.99\% flip rate. Because these simpler alternatives are only tested on a single seed on Qwen3-4B, it is unclear how much of the remaining gap to QGLAS comes from the heuristic quantile-scaling rule (λg=sallβg\lambda_g = s_{\mathrm{all}}\beta_g with 4 hyperparameters βmin⁡=0.3,βmax⁡=0.6,c=0.5,Q100−Q25\beta_{\min}=0.3, \beta_{\max}=0.6, c=0.5, Q_{100}-Q_{25}) vs. single-seed variance and Creative Writing score inflation.

Questions for authors
  1. Assessment-critical (Task-level degradation & scale-normalized retention): Across Table 1 and Table 3, QGLAS consistently lowers both IFBench and Hard Prompts relative to NoBonus (except Hard Prompts under LLM-Judge), while Creative Writing accounts for >90%>90\% of the net macro QGR denominator on Qwen3-4B. What are the per-benchmark retention numbers and z-score-normalized (or NoBonus-normalized) macro scores across Tables 1–3? Why does QGLAS consistently lose ground on IFBench and Hard Prompts while gaining on Creative Writing—is GPT-4.1 on Creative Writing rewarding stylistic density when responses are compressed from ∼4,200\sim 4{,}200 to ∼2,845\sim 2{,}845 tokens?

  2. Clarification affecting soundness (Standardized vs. unnormalized GSPO equations): Section 3.1 (Eqs. 2–3) and Section 3.3 (Eqs. 8–9) derive the method with group standard deviation σr\sigma_r in the denominator, whereas Section 3.3 (lines 229–232) and Appendix A.1 / B.4 (Eq. 20) state that σr\sigma_r normalization is disabled in GSPO (Aiq=ri−rˉA_i^q = r_i - \bar{r}). Were the offline sign-reversal rates in Figure 3a (5.08%,18.56%,11.54%5.08\%, 18.56\%, 11.54\%) and pairwise reordering rates in Appendix Table 9 computed using the unnormalized centered advantages actually used in GSPO, or σr\sigma_r-standardized advantages?

  3. Assessment-critical (Uncertainty for single-seed runs & multi-seed check of GR3 + Zero): Can you provide prompt-level paired bootstrap 95%95\% confidence intervals for all single-seed comparisons in Table 1 (GLM-4.7-Flash), Table 2 (ablations), Table 3 (Rubric and LLM-Judge), and Figure 3b (GR3 + Zero/Sign/Restore), and report 3-seed results on Qwen3-4B for GR3 + Zero and w/o Advantage-Level Shaping?

Reviewer_USmiOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies length control for open-ended reinforcement learning with dense quality rewards. QGLAS leaves quality-only advantage signs unchanged and adds a bounded, one-sided length bonus only to shorter positive-advantage responses. Its strength adapts to the reward spread of each rollout group. Experiments with Qwen3-4B and GLM-4.7-Flash report roughly 32% shorter generations than quality-only RL while retaining approximately all of its macro-averaged quality gain. Comparisons with GR3 and GRLC, sign-correction and component ablations, and additional reward/judge checks support the practical value of the design.

Critical strengths

S1. Clear, inexpensive intervention The positive-only, one-sided bonus offers an interpretable way to separate quality direction from length pressure. The paper appropriately avoids treating preserved advantage signs as a theorem of downstream non-degradation.

S2. Strong complete-method comparisons At similar compression, QGLAS retains substantially more macro quality gain than GR3 and GRLC on both tested model families. This is more informative than reporting shorter responses without controlling the quality trade-off.

S3. Useful checks beyond the main table The sign-correction controls and structural ablations probe the mechanism. Three-seed Qwen results, direct comparisons, length-controlled Bradley–Terry evaluation, and an alternative judge address several obvious sources of evaluation bias.

Critical weaknesses

W1. Adaptive-strength attribution is incomplete The fixed-λ and fixed-spread controls use constants derived from the full method’s training averages. These are useful controls, but they do not establish that adapting strength to the current group is better than a well-tuned constant at the same compression. A fixed-strength sweep or a shuffled-strength control would distinguish conditional adaptation from merely choosing a better strength distribution. This limits the mechanism claim more than the overall method comparison.

W2. Cross-reward superiority has a narrower evidence base Table 3 evaluates alternative reward sources using NoBonus and QGLAS, but does not include GR3/GRLC under those same sources. It establishes retained quality versus uncompressed RL, not comparative superiority across reward models. The GLM results also use one training seed, as disclosed. The broad robustness interpretation should reflect these limits.

W3. Aggregate preservation is not task-wise preservation QGR normalizes a macro-averaged score gain and can conceal capability trade-offs. For GLM, IFBench falls from 53.00 in the base model to 40.11 under NoBonus and 39.00 under QGLAS. Appendix B.3 acknowledges that much of this comes from the quality-only objective. The headline should therefore remain about retaining aggregate RL gains, with task-level losses visible when interpreting deployment value.

Questions for authors

Q1. At matched compression, how does QGLAS compare with a validation-tuned fixed λ and with group-shuffled adaptive strengths? Which part of the gain depends on conditioning strength on the current group?

Q2. Can the authors provide the closest GR3/GRLC comparison under one alternative reward source, or explicitly limit the cross-reward claim to preservation relative to NoBonus?

Q3. What task-level quality constraints would be appropriate alongside QGR, and do the selected operating points satisfy them? This would clarify when aggregate gain retention is an acceptable trade-off.

Reviewer_TaT5OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies length control in RL with graded rewards, such as reward models or LLM judges, where responses grow longer during training. It argues that reward-level shaping methods such as GR3 and GRLC change the group statistics, so they can perturb the quality advantages of all responses and even flip their signs when quality differences within a group are small. On stored rollouts, GR3 flips many more advantages under dense rewards than under binarized ones. The paper then proposes QGLAS, which computes advantages from the quality reward alone. It adds a bounded bonus at the advantage level, only to responses with a positive quality advantage that are shorter than the mean length of these responses, so no advantage changes sign. The bonus strength grows with the overall reward spread in the group and shrinks with the spread among the favored responses. On Qwen3-4B and GLM-4.7-Flash, QGLAS keeps more of the quality gain of quality-only RL than GR3 and GRLC at about 30% compression.

Critical strengths
  1. The method is simple, and it beats both length-control baselines on every benchmark. QGLAS only adds a bounded bonus to the quality advantage of each group, so it is easy to implement. At about 30% compression, it has the highest score among the length-controlled methods on every benchmark for both policy models.

  2. The experiments are extensive. The main comparison on Qwen3-4B is repeated over three training seeds, QGLAS is also trained with two LLM-judge rewards without retuning, and the ablations separate the structural constraints from the adaptive strength. The analysis of ranking reversals is also informative. QGLAS mainly reorders favored responses whose quality advantages are close, while pairs with a large advantage gap are rarely reordered, so quality still dominates when it clearly separates the responses.

Critical weaknesses
  1. Most of the ideas already exist in length-control work for verifiable rewards. DRPO [1] already argues that length should weaken the positive signal of a correct response but never reverse it. Section 3 of [1] also shows that reward-level length penalties can flip the advantage of correct but long responses, and [1] normalizes the length signal only within the positive set. DDCA [2], which the paper cites, already adds a separate length advantage that is computed only within the correct subset. QGLAS mainly replaces correctness with a positive quality advantage, which is exactly the first of the four binarizations in Section 4.3.1. It also keeps only the bonus side and changes the strength statistic. So the new parts of QGLAS seem to be the extension to graded rewards and the adaptive strength rule. The paper does not cite [1], and Section 2 discusses [2] only as a method for verifiable rewards.

  2. The method does not really separate length from quality, and some of its design choices are not tested. Section 3.1 calls the design isolating the length signal, but it only confines the added bonus to the targeted responses. If the reward model or judge already prefers longer responses, as the introduction suggests, the quality advantage itself carries this length preference, and QGLAS keeps its sign. The paper does not measure how strongly its rewards correlate with length. The favored set is simply every response above the group-mean reward, and no other threshold is tried, so it is unclear how its size affects the results. Also, Appendix A.1 says that all experiments disable the standard deviation normalization. So Eq. 8 with σr\sigma_r is never tested, and it is unclear whether QGLAS works as well with the standardized advantage of Eq. 2.

  3. The ablations are not fully described, and the paper does not show what is actually compressed. Among the structural ablations, only the joint one is defined precisely in Appendix B.4. For the single ablations, such as removing one-sided shaping or positive-only gating, the paper does not say how the bonus and the reference length are computed. The scalar shaping strength used to match compression in Table 8 is not defined either. Also, length counts both thinking and answer tokens, but the paper does not show whether QGLAS shortens the reasoning or the visible answer, or how this differs from GR3 and GRLC. It also does not show how the methods behave in the groups where GR3 flips an advantage. The ranking-reversal analysis in Table 9 is reported for QGLAS only, so it is unclear how often the baselines reorder the favored responses.

[1] Li, Gang, et al. "DRPO: Efficient Reasoning via Decoupled Reward Policy Optimization." The Fourteenth International Conference on Learning Representations. 2026.

[2] Peng, Keqin, et al. "Think Dense, Not Long: Dynamic Decoupled Conditional Advantage for Efficient Reasoning." arXiv preprint arXiv:2602.02099 (2026).

Questions for authors
  1. Clarification on W1. Please position QGLAS against [1] and [2]. In particular, please explain what it adds beyond replacing correctness with a positive quality advantage, which is one of the binarizations already used in Section 4.3.1.

  2. Requested comparison for W2. Please report one run with the standard deviation normalization and Eq. 8 with σr\sigma_r, for example on Qwen3-4B with the main reward model. This would show whether QGLAS also works with the standardized advantage of Eq. 2.

  3. Clarification on W2. Please report how strongly the training rewards correlate with response length, and discuss how the size of the favored set, or another choice such as the top quarter of the group, would change the results.

  4. Clarification on W3. Please define each structural ablation precisely, including the bonus, the reference length and the scalar strength used in Table 8.

  5. Suggestion on W3. Please add case studies that split the saved length into thinking and answer tokens for QGLAS, GR3 and GRLC, with examples from groups where GR3 flips an advantage. Please also report the ranking-reversal rates of Table 9 for the two baselines.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal and Review Discussion

Summary of Rebuttal and Review Discussion

Dear Area Chair and Reviewers,

We sincerely thank all reviewers for their constructive feedback and engaging discussions. We summarize the key feedback, additional experiments, and discussion updates below.

1. Strengths Recognized by Reviewers

All three reviewers recognized important strengths of our work:

  • R1 (TaT5): "The method is simple" and "The experiments are extensive," particularly recognizing the consistent benchmark improvements, multi-seed validation, and ranking-reversal analysis.
  • R2 (USmi): Praised the "Clear, inexpensive intervention" and "Strong complete-method comparisons," stating "I have no substantive concerns about the paper's soundness."
  • R3 (ifGD): Recognized the "Clear diagnosis of advantage sign reversals under dense vs. binary rewards" and "Clean structural ablations and consistent 3-seed gains."

2. Key Concerns Addressed During Rebuttal

We conducted additional experiments and analyses to address the reviewers' concerns:

  • Novelty and methodology (R1): Expanded comparisons with DRPO and DDCA, clarifying QGLAS's contributions under graded feedback, including advantage polarity preservation, unchanged untargeted advantages, and adaptive shaping strength.
  • Experimental robustness (R1–R3): Added normalized-advantage experiments (99.5% QGR at 34.3% compression), fixed/shuffled-strength controls (81.2% vs. 102.0% QGR at similar compression), cross-reward comparisons, and additional multi-seed validation.
  • Evaluation and claim scope (R2, R3): Added task-level and z-normalized evaluations, clarified the interpretation of aggregate QGR, and revised the abstract and conclusion to emphasize matched-compression superiority rather than implying task-wise quality preservation.

3. Discussion Updates

  • R1 (TaT5): The reviewer increased their score from 2 to 3, explicitly stating:

    "Thank you for the detailed response and the new results, which have resolved my concerns. Given the quality of this work, I raise my score from 2 to 3."

  • R2 (USmi): The reviewer gave an initial score of 3 and recognized the method's soundness, stating: "I have no substantive concerns about the paper's soundness." We addressed all three questions with additional experiments and clarifications. No further response has been received from the reviewer so far.

  • R3 (ifGD): The reviewer acknowledged and corrected several issues in the original assessment, including incorrect numerical results and an overgeneralized claim regarding task-level degradation across models and reward sources.

    Importantly, the reviewer subsequently recognized:

    "QGLAS is a simple method with a clear advantage over GR3 and GRLC at matched compression, and the new results make that comparison more convincing."

    The reviewer then identified two specific additions:

    "Two targeted additions would strengthen the paper. The first is the w/o Advantage-Level ablation on seeds 43 and 44, so that it can be compared with QGLAS as in Table 6. The second is the number of training seeds behind the fixed-λ sweep and the group-shuffled λ control."

    We have addressed both requests:

    • Completed the w/o Advantage-Level ablation on seeds 43 and 44. QGLAS outperformed the reward-level variant on all three matched seeds, with an average improvement of 1.43 macro-score points at similar response lengths.
    • Explicitly clarified that each fixed-λ configuration and the group-shuffled-λ control used one training seed (seed 42).

    Our latest response has addressed both requests. The discussion is still ongoing, and no further reply from the reviewer has been received so far.

All corresponding manuscript revisions are highlighted in blue in the updated PDF.

We sincerely appreciate the reviewers' thoughtful feedback and constructive discussions. Their suggestions have helped us strengthen the experimental evidence, clarify the methodological contributions, and improve the presentation of our work.

Thank you all for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.