- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies response-length control in open-ended LLM reinforcement learning, arguing that under dense, graded quality rewards, adding length bonuses at the reward level can flip the sign of group-relative advantages for responses near the group mean. The authors propose Quality-Gated Length Advantage Shaping (QGLAS, Section 3), which modifies advantages directly via (Eq. 1): it restricts length bonuses to the quality-favored subset (positive-only gating), assigns bounded one-sided bonuses only to responses shorter than the positive-set average length (Eq. 4), and scales the group bonus using the overall reward spread and the positive-set spread (Eqs. 6–9) so that (Eq. 5).
QGLAS is tested under 1,000-step GSPO training on 13K open-ended prompts (Skywork-Reward-V2-Llama-3.1-8B) with Qwen3-4B and GLM-4.7-Flash (30B-A3B), evaluated on IFBench, Arena-Hard-v2 Hard Prompts, and Arena-Hard-v2 Creative Writing (Section 4, Table 1). While the paper reports headline macro-average Quality-Gain Retention (QGR, Eq. 10) of 102.0% (Qwen3-4B) and 98.4% (GLM-4.7-Flash) at roughly 32% length compression—alongside offline sign-flip diagnostics (Figure 3a) and ablations (Table 2)—a closer inspection of Table 1 reveals that this "near-lossless" macro QGR is heavily distorted by benchmark scale imbalance (Creative Writing accounts for over 91% of the net denominator on Qwen3-4B, while both IFBench and Hard Prompts actually degrade relative to NoBonus), alongside a mismatch between the standardized advantage theory in Section 3.1/3.3 and the unnormalized GSPO implementation and single-seed reporting for GLM-4.7-Flash and alternative rewards.
Critical strengths
1. Clear diagnosis of advantage sign reversals under dense vs. binary rewards.
Section 4.3 (Figure 3a–b) identifies a concrete reason why reward-level length mixing (, GRLC) degrades open-ended RL more severely than binary RLVR: dense rewards cluster near the group mean , causing the group-mean shift from length bonuses to flip the sign of of rollout advantages, and manually overriding just those flipped signs during training recovers a large fraction of the lost score ( QGR under Zero in Figure 3b).
2. Clean structural ablations and consistent 3-seed gains over and GRLC on Qwen3-4B.
Within the Qwen3-4B learned-reward setting, the matched-compression comparison against and GRLC is backed by 3 training seeds (Appendix Table 6: macro score vs. and ), and Table 2 (along with compression-matched Appendix Table 8) systematically ablates advantage-level shaping, positive-only gating, one-sided shaping, and adaptive scaling.
Critical weaknesses
1. The headline "near-lossless () quality retention" claim is an artifact of unweighted macro-averaging dominated by Creative Writing, while objective instruction following (IFBench) and Hard Prompts both degrade.
As noted under Soundness, the central empirical claim of the paper relies on the aggregate QGR metric in Eq. (10). However, inspecting the individual benchmark columns in Table 1 tells a very different story:
- On
Qwen3-4B, quality-only RL (NoBonus) already degrades strict instruction following onIFBenchby points relative toBase(), and QGLAS degrades it further to ( vs.NoBonus, vs.Base). OnHard Prompts, QGLAS also drops relative toNoBonus( vs. , retaining of the gain). The only reason QGLAS's macroQGRexceeds () is thatCreative Writingleaps by points fromBase() toNoBonus()—accounting for of the entire macroQGRdenominator—and QGLAS scores () on that single GPT-4.1-judged benchmark. - On
GLM-4.7-Flash, the exact same pattern occurs:IFBenchplummets by points fromBase() toNoBonus() and drops another points under QGLAS (), whileHard Promptsalso falls from (NoBonus) to (QGLAS, ). - Even in Table 3 (
Rubric-based JudgeonQwen3-4B),IFBenchdrops from (Base) to (NoBonus) and (QGLAS, vs.NoBonus), whileHard Promptsdrops from (NoBonus) to (QGLAS, ), yet the paper still reports a high macroQGR() becauseCreative Writingrose from to . Because an unweighted average where one subjective creative-writing benchmark carries of the denominator weight masks consistent regressions onIFBenchandHard Promptsacross every model and reward source, the current metric and framing significantly overstate across-the-board quality preservation. This directly drives my weak-reject rating.
2. Missing statistical uncertainty for GLM-4.7-Flash, alternative reward sources (Table 3), and causal interventions (Figure 3b), plus incomplete comparison to simple sign-corrected baselines.
While the Qwen3-4B learned-RM main row uses 3 seeds, GLM-4.7-Flash (Table 1, trained once on seed 42 and merely re-evaluated 3 times, Appendix A.1 lines 681–683), the Rubric and LLM-as-a-Judge experiments in Table 3, the targeted sign-reversal fixes in Figure 3b, and the Pareto sweep points in Figure 1 / Appendix Table 7 are all single-seed runs without prompt-level bootstrap confidence intervals. Furthermore, Figure 3b shows that simply zeroing out sign-flipped advantages in (GR3 + Zero) already jumps QGR from to at the same compression, and in Table 2, QGLAS w/o Advantage-Level Shaping (which keeps positive-only one-sided bonuses at the reward level) reaches QGR at CR with only a flip rate. Because these simpler alternatives are only tested on a single seed on Qwen3-4B, it is unclear how much of the remaining gap to QGLAS comes from the heuristic quantile-scaling rule ( with 4 hyperparameters ) vs. single-seed variance and Creative Writing score inflation.
Questions for authors
-
Assessment-critical (Task-level degradation & scale-normalized retention): Across Table 1 and Table 3, QGLAS consistently lowers both
IFBenchandHard Promptsrelative toNoBonus(exceptHard Promptsunder LLM-Judge), whileCreative Writingaccounts for of the net macroQGRdenominator onQwen3-4B. What are the per-benchmark retention numbers and z-score-normalized (orNoBonus-normalized) macro scores across Tables 1–3? Why does QGLAS consistently lose ground onIFBenchandHard Promptswhile gaining onCreative Writing—is GPT-4.1 onCreative Writingrewarding stylistic density when responses are compressed from to tokens? -
Clarification affecting soundness (Standardized vs. unnormalized GSPO equations): Section 3.1 (Eqs. 2–3) and Section 3.3 (Eqs. 8–9) derive the method with group standard deviation in the denominator, whereas Section 3.3 (lines 229–232) and Appendix A.1 / B.4 (Eq. 20) state that normalization is disabled in GSPO (). Were the offline sign-reversal rates in Figure 3a () and pairwise reordering rates in Appendix Table 9 computed using the unnormalized centered advantages actually used in GSPO, or -standardized advantages?
-
Assessment-critical (Uncertainty for single-seed runs & multi-seed check of
GR3 + Zero): Can you provide prompt-level paired bootstrap confidence intervals for all single-seed comparisons in Table 1 (GLM-4.7-Flash), Table 2 (ablations), Table 3 (Rubric and LLM-Judge), and Figure 3b (GR3 + Zero/Sign/Restore), and report 3-seed results onQwen3-4BforGR3 + Zeroandw/o Advantage-Level Shaping?
Summary of Rebuttal and Review Discussion
Summary of Rebuttal and Review Discussion
Dear Area Chair and Reviewers,
We sincerely thank all reviewers for their constructive feedback and engaging discussions. We summarize the key feedback, additional experiments, and discussion updates below.
1. Strengths Recognized by Reviewers
All three reviewers recognized important strengths of our work:
2. Key Concerns Addressed During Rebuttal
We conducted additional experiments and analyses to address the reviewers' concerns:
3. Discussion Updates
R1 (TaT5): The reviewer increased their score from 2 to 3, explicitly stating:
R2 (USmi): The reviewer gave an initial score of 3 and recognized the method's soundness, stating: "I have no substantive concerns about the paper's soundness." We addressed all three questions with additional experiments and clarifications. No further response has been received from the reviewer so far.
R3 (ifGD): The reviewer acknowledged and corrected several issues in the original assessment, including incorrect numerical results and an overgeneralized claim regarding task-level degradation across models and reward sources.
Importantly, the reviewer subsequently recognized:
The reviewer then identified two specific additions:
We have addressed both requests:
Our latest response has addressed both requests. The discussion is still ongoing, and no further reply from the reviewer has been received so far.
All corresponding manuscript revisions are highlighted in blue in the updated PDF.
We sincerely appreciate the reviewers' thoughtful feedback and constructive discussions. Their suggestions have helped us strengthen the experimental evidence, clarify the methodological contributions, and improve the presentation of our work.
Thank you all for your time and consideration.