- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper considers the setting of RL post-training via group-based policy gradient methods such as GRPO or RLOO. It therein proposes to consider not just the success rate per prompt, but the total absolute advantage (i.e. , with indexing answers to a given prompt). The paper shows under which conditions this total absolute advantage is maximized, and proposes to use it to select training prompts. Experiments then show a benefit of variance based prompt selection.
Critical strengths
- Prompt selection is a useful topic
Critical weaknesses
- The proposed metric, total absolute advantage, is motivated by its connection to the gradient norm. However, the paper does not show that gradient norm is a useful target by itself. The gradient could have a large norm, while being inaccurate, for example pointing in a different direction from the full gradient
- The evaluated method simply selects prompts by their reward variance. The paper itself admits that this is not novel, thus, even though it works, I do not see how these results support the usage of total absolute advantage as a metric.
- Total absolute advantage is not shown to be advantageous to select a "warm-up", i.e. SFT setup. While the experiments show that total absolute advantage decreases with increased SFT epochs, it is not shown that this is helpful for downstream RL performance. This is particularly suprising, as the introduction extensively discusses warm-up.
Questions for authors
Is it possible to show a connection between "gradient efficiency" and the actual accuracy of the estimated gradient vs the true gradient? Even just an empirical investigation would significantly strengthen the support for the proposed metric.
5 comments
The rebuttal does not change my concerns.
The rebuttals are also clearly pure AI slop. This is very disrespectful to the reviewers, and I would respectfully advice the authors against doing this in future submissions.
I am keeping my score.
Thank you for your service as a reviewer. However, dismissing a technical rebuttal as "pure AI slop" without engaging with any of its derivations or experimental results is neither accurate nor constructive.
- ICLR 2027 Policy: Under the official ICLR 2027 AI Policy for Authors, using LLMs to assist with drafting and editing for readability is explicitly permitted, with the authors taking full responsibility for the scientific content and accuracy. Using an LLM for prose or editing does not make a response "AI slop."
- Substance of the rebuttal: Every claim in our response is backed by newly executed experiments and verified proofs in the revised manuscript—including the exact per-prompt gradient identity (Proposition 3, verified by exact enumeration to ), the held-out RLOO gradient-alignment probes you specifically requested (Table 1, Section 5.1, Appendix E), and the multi-draw/DAPO comparisons (Tables 10–11).
If there is a specific technical flaw in any derivation or experiment we added, we welcome a concrete critique of the substance rather than a blanket dismissal of the writing style.
Response to Reviewer znrf
Thank you for a sharp review. Your question names exactly the experiment we needed, and we have run it.
W1 and Question. Norm versus direction, and the estimated vs the true gradient. We agree that a norm argument alone is insufficient, and we address it in two steps.
Theory (Prop. 3). The mass is tied to the true gradient, not only to an estimator's norm. For a binary reward the exact per-prompt gradient is ∇p(x)=p(1−p)Δx, with Δx=μ+−μ−. The RLOO group gradient is unbiased for it, and one expected step on a set S changes a target objective JT to first order by α⟨∇JS,∇JT⟩=∣S∣α∑x∈Sp(1−p)⟨Δx,∇JT⟩. So η is the weight each prompt carries in ∇J itself, and your direction question reduces to whether Δx aligns with ∇JT.
Measurement (Sec. 5.1, App. E). We estimated the true gradient of held-out success with the unbiased RLOO estimator at G=64 on the 256 held-out problems. Its two independent halves agree at cosine 0.62 to 0.69, our noise ceiling. We compared it with 200 training-sized minibatch updates from each of six prompt sets. In every pair, updates from the high-η set point further along the held-out gradient, by +0.057 in cosine at Countdown 3B, +0.068 at GSM8K 7B, and +0.056 for fresh against a warm-up's own prompts, with intervals excluding zero. On the same probes, a group's gradient norm tracks its mass (ρ = 0.55 to 0.68), and ∥Δx∥ is flat in p (bin medians within 1.38x). That flatness is the condition under which η measures the expected magnitude of the true gradient. The predicted improvement ratio matches the realized early gain at Countdown 3B (1.85x vs 1.85x) and understates it on GSM8K (1.97x vs 3.90x). Along training the alignment advantage is front-loaded and reverses by step 100, moving with in-run η (ρ=0.86).
W2. Selection by reward variance is not novel, so these results do not support the metric. We agree on the first half, and the revision says so in the contributions. On the second, the selection experiment was never the only support. The metric makes other predictions that are not about selection, and each is tested.
W3. Not shown helpful for choosing a warm-up. You identified a real mismatch between our introduction and our results, and we have fixed the introduction. The paper does not claim η selects warm-ups. It reports that it does not, and the introduction now says so before the results. What the paper claims about warm-ups is narrower. Training one on its own RL prompts removes the gradient they would supply. That is certain (η 0.061 vs 0.196 on fresh prompts at 3B, 0.011 vs 0.175 at 7B), and the gradient probes confirm it in the update itself (+0.056 in held-out alignment, a 6.45x predicted-improvement ratio). Whether it also costs held-out accuracy is not established. Over five seeds the overlapping arm loses 0.013 while a matched fresh arm gains 0.006, a difference positive on four of five seeds with p=0.064. The revision states this as unresolved in the discussion rather than as a finding.
Presentation. We restructured around one central claim, cut the instrument section to two sentences, and moved five secondary paragraphs to the appendix. Each results subsection now tests one consequence of the claim, as outlined in our response to Reviewer up3Q.
Summary of the method. Your summary describes the method as selecting by total absolute advantage. The metric is the expected normalized advantage mass of a policy over a prompt distribution (Eq. 4). Its closed form contains a term, Varx(p), that no per-prompt score has. Selection is one of two interventions on it, and the other, the warm-up overlap, moves it the opposite way.