acceptodds
Under review as a conference paper at ICLR 2027

Matched Compute Is Not Matched Signal: Where the Gradient Goes in Group-Relative Reinforcement Learning

Abstract

Reinforcement learning from verifiable rewards is usually preceded by a supervised warm-up, and runs are compared at matched compute. But group-relative methods such as GRPO learn only from prompts whose sampled answers disagree: a group of rollouts that all earn the same reward contributes exactly zero gradient, however much it cost to generate. Two runs with the same budget can therefore receive very different amounts of learning signal. Prior work uses this to filter prompts during training; we make it exact and extend it to policies and prompt sets. We derive the gradient signal a group carries under GRPO, Dr.GRPO and RLOO and average it over the prompts. The result is a signal efficiency , where is mean accuracy, is a prompt's success probability and depends only on the group size. A more accurate policy can therefore carry less signal. Across 32 runs given identical budgets, the generations that actually carried a gradient differed by a factor of . A warm-up trained on the problems its reinforcement-learning phase will use silences them: after epochs, solution distillation solves of them and falls from to . The same accounting explains why a base model barely improves: its gradient is real, but almost all of it comes from the format reward rather than from being right. Choosing prompts by , from a single sampling pass before training, beats a random set of the same size and budget on all runs across four settings, a search task and GSM8K at 3B to 14B, by to points of held-out accuracy. The lead survives at twice the training horizon, though its signal advantage fades as the chosen prompts are solved, and reverses on every seed in the one setting where the random control climbs far enough toward . Two plausible variants of the rule each fail in some setting, by overshooting an optimum the accounting locates. What does not do is rank warm-ups or predict how much a run will gain: it prices prompts, not checkpoints. Runs should therefore be compared by the generations that carry a gradient, not the generations spent. Code: https://anonymous.4open.science/r/signal-efficiency-rl-D5C1.

Primary area:
reinforcement learning
Keywords:
reinforcement learning from verifiable rewards, GRPO, Dr.GRPO, RLOO, supervised warm-up, advantage mass, reward variance, prompt selection, group-relative policy optimization

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_znrfOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper considers the setting of RL post-training via group-based policy gradient methods such as GRPO or RLOO. It therein proposes to consider not just the success rate per prompt, but the total absolute advantage (i.e. ∑i∣Ai∣\sum_i |A_i|, with ii indexing answers to a given prompt). The paper shows under which conditions this total absolute advantage is maximized, and proposes to use it to select training prompts. Experiments then show a benefit of variance based prompt selection.

Critical strengths
  • Prompt selection is a useful topic
Critical weaknesses
  • The proposed metric, total absolute advantage, is motivated by its connection to the gradient norm. However, the paper does not show that gradient norm is a useful target by itself. The gradient could have a large norm, while being inaccurate, for example pointing in a different direction from the full gradient ∇θ[J(θ)]\nabla_\theta [ J(\theta)]
  • The evaluated method simply selects prompts by their reward variance. The paper itself admits that this is not novel, thus, even though it works, I do not see how these results support the usage of total absolute advantage as a metric.
  • Total absolute advantage is not shown to be advantageous to select a "warm-up", i.e. SFT setup. While the experiments show that total absolute advantage decreases with increased SFT epochs, it is not shown that this is helpful for downstream RL performance. This is particularly suprising, as the introduction extensively discusses warm-up.
Questions for authors

Is it possible to show a connection between "gradient efficiency" and the actual accuracy of the estimated gradient vs the true gradient? Even just an empirical investigation would significantly strengthen the support for the proposed metric.

5 comments
AuthorsOpenReview ↗

Response to Reviewer znrf

Thank you for a sharp review. Your question names exactly the experiment we needed, and we have run it.

W1 and Question. Norm versus direction, and the estimated vs the true gradient. We agree that a norm argument alone is insufficient, and we address it in two steps.

Theory (Prop. 3). The mass is tied to the true gradient, not only to an estimator's norm. For a binary reward the exact per-prompt gradient is ∇p(x)=p(1−p)Δx\nabla p(x) = p(1-p)\Delta_x, with Δx=μ+−μ−\Delta_x = \mu^+ - \mu^-. The RLOO group gradient is unbiased for it, and one expected step on a set SS changes a target objective JTJ_T to first order by α⟨∇JS,∇JT⟩=α∣S∣∑x∈Sp(1−p)⟨Δx,∇JT⟩\alpha\langle \nabla J_S, \nabla J_T \rangle = \frac{\alpha}{|S|}\sum_{x\in S} p(1-p)\langle \Delta_x, \nabla J_T\rangle. So η\eta is the weight each prompt carries in ∇J\nabla J itself, and your direction question reduces to whether Δx\Delta_x aligns with ∇JT\nabla J_T.

Measurement (Sec. 5.1, App. E). We estimated the true gradient of held-out success with the unbiased RLOO estimator at G=64G = 64 on the 256 held-out problems. Its two independent halves agree at cosine 0.62 to 0.69, our noise ceiling. We compared it with 200 training-sized minibatch updates from each of six prompt sets. In every pair, updates from the high-η\eta set point further along the held-out gradient, by +0.057 in cosine at Countdown 3B, +0.068 at GSM8K 7B, and +0.056 for fresh against a warm-up's own prompts, with intervals excluding zero. On the same probes, a group's gradient norm tracks its mass (ρ\rho = 0.55 to 0.68), and ∥Δx∥\|\Delta_x\| is flat in pp (bin medians within 1.38x). That flatness is the condition under which η\eta measures the expected magnitude of the true gradient. The predicted improvement ratio matches the realized early gain at Countdown 3B (1.85x vs 1.85x) and understates it on GSM8K (1.97x vs 3.90x). Along training the alignment advantage is front-loaded and reverses by step 100, moving with in-run η\eta (ρ=0.86\rho = 0.86).

W2. Selection by reward variance is not novel, so these results do not support the metric. We agree on the first half, and the revision says so in the contributions. On the second, the selection experiment was never the only support. The metric makes other predictions that are not about selection, and each is tested.

  • The collapse when a warm-up trains on its RL prompts (η\eta 0.600 to 0.065 with held-out accuracy at its highest).
  • The base-model bound, via the cancellation of a shaping term's weight under GRPO.
  • The reversal predicted only where the control starts below p=1/2p = 1/2.
  • The direct gradient measurements you asked for.

W3. Not shown helpful for choosing a warm-up. You identified a real mismatch between our introduction and our results, and we have fixed the introduction. The paper does not claim η\eta selects warm-ups. It reports that it does not, and the introduction now says so before the results. What the paper claims about warm-ups is narrower. Training one on its own RL prompts removes the gradient they would supply. That is certain (η\eta 0.061 vs 0.196 on fresh prompts at 3B, 0.011 vs 0.175 at 7B), and the gradient probes confirm it in the update itself (+0.056 in held-out alignment, a 6.45x predicted-improvement ratio). Whether it also costs held-out accuracy is not established. Over five seeds the overlapping arm loses 0.013 while a matched fresh arm gains 0.006, a difference positive on four of five seeds with p=0.064p = 0.064. The revision states this as unresolved in the discussion rather than as a finding.

Presentation. We restructured around one central claim, cut the instrument section to two sentences, and moved five secondary paragraphs to the appendix. Each results subsection now tests one consequence of the claim, as outlined in our response to Reviewer up3Q.

Summary of the method. Your summary describes the method as selecting by total absolute advantage. The metric is the expected normalized advantage mass of a policy over a prompt distribution (Eq. 4). Its closed form contains a term, Varx(p)\mathrm{Var}_x(p), that no per-prompt score has. Selection is one of two interventions on it, and the other, the warm-up overlap, moves it the opposite way.

The rebuttal does not change my concerns.

The rebuttals are also clearly pure AI slop. This is very disrespectful to the reviewers, and I would respectfully advice the authors against doing this in future submissions.

I am keeping my score.

Thank you for your service as a reviewer. However, dismissing a technical rebuttal as "pure AI slop" without engaging with any of its derivations or experimental results is neither accurate nor constructive.

  1. ICLR 2027 Policy: Under the official ICLR 2027 AI Policy for Authors, using LLMs to assist with drafting and editing for readability is explicitly permitted, with the authors taking full responsibility for the scientific content and accuracy. Using an LLM for prose or editing does not make a response "AI slop."
  2. Substance of the rebuttal: Every claim in our response is backed by newly executed experiments and verified proofs in the revised manuscript—including the exact per-prompt gradient identity (Proposition 3, verified by exact enumeration to 1.8×10−151.8 \times 10^{-15}), the held-out G=64G=64 RLOO gradient-alignment probes you specifically requested (Table 1, Section 5.1, Appendix E), and the multi-draw/DAPO comparisons (Tables 10–11).

If there is a specific technical flaw in any derivation or experiment we added, we welcome a concrete critique of the substance rather than a blanket dismissal of the writing style.

Reviewer_up3QOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the mismatch between computational budget and effective learning signals in group-relative reinforcement learning, deriving a signal efficiency metric based on the distribution of per-prompt success probabilities. Building on this analysis, the authors propose a difficulty-based prompt selection strategy that improves performance over random selection, while acknowledging that signal efficiency does not reliably predict downstream RL improvements.

Critical strengths
  • The paper provides a systematic empirical analysis across multiple model scales and tasks, with extensive ablation studies and careful reporting of both positive and negative results.
  • The question this paper investigated (how to post-train LLMs with the most learning signal) is an important problem in current LLM RL post training
Critical weaknesses
  1. Limited novelty. The main insights, including reward variance-based learning signals and difficulty-based prompt selection, are largely established in prior work.

  2. Unclear presentation. The paper makes numerous claims without a coherent central narrative, making it difficult to identify its primary contribution.

Questions for authors
  1. Could the authors clarify what new insights the proposed framework provides beyond existing work on reward variance and difficulty-based prompt selection?

  2. Given that signal efficiency does not reliably predict downstream RL improvement, what is the practical value of this metric, and why should it be interpreted as effective compute?

4 comments
AuthorsOpenReview ↗

Response to Reviewer up3Q

Thank you. We agree with both weaknesses as stated and have changed the paper accordingly.

W1/Q1. Novelty beyond reward-variance and difficulty-based selection. Per-prompt reward variance as a learnability signal, and selection by difficulty, are established. We cite DAPO, GRESO, VCRL and Bae et al. for exactly that, and the revised contributions now list them as known. What is new follows from aggregating that quantity over a policy and using it to audit real budgets. None of the following is in the work we cite.

  1. The policy-level decomposition η=cG[a(1−a)−Varx(p)]\eta = c_G[a(1-a) - \mathrm{Var}_x(p)], in which accuracy and the spread of difficulty compete, so a more accurate policy can supply less gradient (Thm. 1, Cor. 1).
  2. The identity ∇p=p(1−p)(μ+−μ−)\nabla p = p(1-p)(\mu^+-\mu^-) that makes η\eta the weight on the true gradient (Prop. 3). We confirm it on real checkpoints, where gradient norm tracks mass and high-η\eta updates align better with the held-out gradient (Sec. 5.1).
  3. A hazard of GRPO's normalization. A shaping term's weight cancels on exactly the groups where correctness is silent, which explains a base model's flat curve as a full-magnitude format gradient (Sec. 5.4).
  4. A warm-up trained on its own RL prompts silences them, with η\eta falling from 0.600 to 0.065 while held-out accuracy is at its highest (Sec. 5.3).
  5. A selected set spends itself (Prop. 5). A reversal is predicted only where the control starts below p=1/2p = 1/2, and occurs only there. The gradient alignment follows the same course (Table 4).
  6. Across 32 runs at identical nominal budgets, the generations carrying a gradient differ 8.8-fold.

The p(1−p)p(1-p) rule is not on this list. It is the instrument we use to move η\eta at fixed weights. We now compare it with DAPO, a difficulty band and online filtering at equal total cost (Table 11), and it is level with DAPO.

W2. No coherent central narrative. This was fair. The revision states one claim at the top of the introduction. The gradient a fixed budget buys is set by the policy's outcome variance on its prompt set, it can be measured before training, and two routine decisions change once it is measured. Each results subsection now tests one consequence of it. Sec. 5.1 asks whether the accounting describes the real gradient, Sec. 5.2 covers choosing the prompt set, Sec. 5.3 covers losing it to the warm-up, and Sec. 5.4 states what the accounting explains and cannot predict. The instrument section is reduced to two sentences, and five secondary paragraphs moved to the appendix.

Q2. Practical value, and why "effective compute". It is measurable from one sampling pass before training. It tells a practitioner how much of a budget will produce any correctness gradient, 312 of 4,800 generations for an overtrained warm-up and none at all for one base-model run. We call ηN\eta N effective compute because, by Prop. 3, it counts generations by the weight they put on the true gradient. We do not call it that because it forecasts accuracy, and the revision states the definition explicitly (Sec. 3.2). Empirically, matching effective rather than nominal compute equalizes selected and random arms early in training, where the nominal match leaves a gap of +0.020 to +0.059. Over a whole run it does not, because a selected set spends its signal early (Prop. 5). We have removed wording that implied a forecast.

Thanks for the response and the additional experiments. The new gradient alignment results are helpful, but I still have concerns about the novelty and practical significance of the work.

I also have a concern about Proposition 5. In particular, I don't see how the assumption that per-prompt success probabilities increase monotonically implies the claimed ordering of crossing times or the reversal in signal efficiency. This seems to require additional assumptions about how quickly different prompts are learned.

Overall, while the revision addresses some of my comments, I don't think it resolves my main concerns. I will keep my original score.

AuthorsOpenReview ↗

Thank you for your follow-up

Thank you for taking the time to read our rebuttal and revision, and for your thoughtful follow-up feedback throughout the review process.

AuthorsOpenReview ↗

Follow-up to Reviewer up3Q

Thank you for the follow-up and for finding the gradient-alignment results helpful.

Your reading of Proposition 5 is correct, and we have revised it. Monotone progress gives only the per-prompt rise and fall. The ordering of crossing times now rests on an explicit order-preservation assumption, under which it follows directly. The reversal is stated under an explicit idealization, a common shift in logit space, and it says a lead can be lost only when the control starts further below p=1/2p = 1/2 than the selected set. We tested both assumptions on per-prompt training records (App. F). Order is preserved across starting-difficulty bins throughout training, while rates of learning differ between prompts, so the paper now presents the proposition as saying where a reversal can occur rather than when.

We understand that the novelty and significance concerns remain, and we do not want to repeat our earlier response. We would only add that the claims are now narrower and each is stated with the evidence it rests on.

Reviewer_LHuqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies why reinforcement learning with verifiable rewards can achieve different outcomes under the same nominal compute budget. The authors observe that GRPO and related group-relative estimators produce zero advantages when all sampled responses receive identical rewards, meaning that these rollouts contribute no policy-gradient update through the advantage-weighted objective. Experiments examine supervised warm-up effects, prompt selection, group size, base-model RL, and cross-geometry transfer.

Critical strengths
  1. The paper highlights that equal generation budgets do not necessarily provide equal learning signals. The analysis of supervised warm-ups and prompt overlap further demonstrates that the informativeness of a fixed prompt set can change substantially as the policy improves.
  2. The derivation of closed-form advantage mass for GRPO, Dr.GRPO, and RLOO provides a principled account of how group reward configurations affect the signal available to learning.
  3. The decomposition of policy-level signal efficiency into terms involving mean accuracy and prompt-level variance is particularly useful because it explains why policies with higher accuracy do not necessarily provide more learning signal.
  4. The empirical study includes meaningful controls and negative results. The prompt-selection experiments compare the proposed rule with random selection under matched prompt-set sizes and training budgets.
Critical weaknesses
  1. The proposed metric quantifies absolute advantage mass, but does not directly measure policy-gradient norms or gradient directions. It therefore remains unclear how closely the metric reflects the effective updates that drive learning.
  2. The p(1-p) criterion is related to existing difficulty- and reward-variance-based selection methods. The paper's theoretical analysis is valuable, but the practical advantage over these alternatives is not fully established.
  3. The four settings show consistent gains, but most comparisons reuse a single selected set and a single random control set across seeds. The additional control-set redraw experiment covers only Countdown 3B, leaving the robustness of the results across independently sampled prompt sets uncertain.
  4. The link between signal efficiency and learning performance remains unclear. The proposed measure does not consistently predict downstream gains, and the group-size experiments are inconclusive.
Questions for authors
  1. Have the authors measured actual policy-gradient norms or gradient directions to validate whether the proposed signal-efficiency metric correlates with effective policy updates?
  2. How does selecting prompts by p(1-p) compare with existing difficulty- or reward-variance-based selection methods under the same training budget and evaluation protocol?
  3. Given the inconclusive group-size results and the limited relationship between signal efficiency and final performance, what evidence supports using the proposed metric to predict or improve downstream RL performance?
  4. The authors should clarify whether the observed gains generalize beyond the Countdown 3B control sets or may depend on the particular prompt sets used in the evaluation.
4 comments
AuthorsOpenReview ↗

Response to Reviewer LHuq

Thank you for a constructive review. You named the strengths we most wanted credited, and your four questions shaped the revision.

W1/Q1. Policy-gradient norms and directions. We did both.

Theory. Prop. 3 shows ∇p(x)=p(1−p)Δx\nabla p(x) = p(1-p)\Delta_x with Δx=μ+−μ−\Delta_x = \mu^+ - \mu^-, so η\eta is the mean weight prompts receive in the true gradient. It equals the expected gradient magnitude exactly when ∥Δx∥\|\Delta_x\| does not covary with pp, and direction reduces to how Δx\Delta_x aligns with a target gradient. All identities are checked by exact enumeration (max error 1.8e-15, App. A).

Norms. With gradients of the trainer's own loss on the adapter parameters (correctness reward, no KL), per-group gradient norm tracks normalized mass at Spearman 0.55 to 0.68 in all six sets. After controlling for completion length it is 0.52 to 0.64 (Table 3). The contrast ∥Δx∥\|\Delta_x\|, estimated without bias from two independent groups of 32, is flat across eight difficulty bins, with ∣ρ∣≤0.14|\rho| \le 0.14 and bin medians within 1.38x.

Directions. Against an unbiased RLOO estimate of the held-out gradient (G=64G=64, two halves agreeing at cosine 0.62 to 0.69), updates from the high-η\eta set are better aligned in all three pairs. The cosine differences are +0.057 at Countdown 3B, +0.068 at GSM8K 7B and +0.056 for fresh against a warm-up's own prompts, all with bootstrap intervals excluding zero (Table 1). At Countdown 3B the first-order predicted improvement ratio, 1.85x, matches the realized first-quarter gain ratio, 1.85x. On GSM8K the prediction is 1.97x against a realized 3.90x, so first order understates the advantage there, and we report both. Along training (Table 4), the advantage is largest at step 0, shrinks by step 50 and reverses by step 100. It moves with in-run η\eta at ρ=0.86\rho = 0.86. Separately, the trainer's own logged gradient norms move with in-run η\eta within runs, with median Spearman 0.40 after controlling for length and KL, positive in 372 of 425 runs (App. E).

W2/Q2. Comparison with existing selection methods. Table 11 compares, at Countdown 3B and GSM8K 7B with five seeds, the rule from a full and a cheap pass against dropping silent prompts, p(1−p)2p(1-p)^2, a 0.2≤p^≤0.80.2 \le \hat p \le 0.8 band, online refilling every 50 steps, and DAPO's dynamic sampling. Every selection, probe and discarded rollout is charged. At equal total generations the cheap-pass rule is level with DAPO (0.2062 vs 0.2070 at 3B, 0.8516 vs 0.8492 on GSM8K). It is level with the band at 3B and ahead on GSM8K (0.8516 vs 0.8422), and ahead of online refilling in both settings. Given the same training generations without being charged for discarded rollouts, DAPO edges ahead (0.2164 and 0.8570). That is the exchange the accounting describes, since DAPO buys signal with generations the static rule does not spend. Two variants of our rule beat it at 3B and fail at 7B (Table 9). No fixed rule dominates, and we do not claim p(1−p)p(1-p) is the best selector.

W3/Q4. Robustness across independently drawn sets. We redrew in every setting (Table 10), adding four more random controls and two more selected sets, each selected set built from an independent sampling pass on a disjoint half of the pool. In all four settings each of the three selected sets scores above each of the five controls. Under exchangeability that happens with probability 1/56 per setting and 1.0×10−71.0\times10^{-7} over the four. The effect averaged over draws is +0.027, +0.026, +0.078 and +0.032. Between-draw variance is 32 to 43% of the control's, which is why we now treat the draw, not the seed, as the unit.

W4/Q3. η\eta and downstream performance, and the group-size ladder. We agree η\eta does not predict a run's final gain, and we report that as a result (Sec. 5.4). What supports using it is that it describes the update (Prop. 3, Sec. 5.1) and prices the decisions that change it. Choosing prompts (Sec. 5.2) raises effective compute and accuracy in every setting. A warm-up that has solved its RL prompts (Sec. 5.3) turns 4,800 generations into 312. At matched effective compute early in training the selected and random arms are level (+0.002, +0.003), while at matched nominal compute they are not (+0.020, +0.059). The group-size ladder, we found, ran at the inherited learning rate of 10−610^{-6}, which App. H.5 shows confines held-out accuracy to a channel about 0.02 wide. Its flat outcome is therefore weak evidence either way, and the revision says so (App. I). The mass accounting it was designed to test is confirmed.

Thank you for the detailed response and additional experiments. Most of my concerns have been addressed. I still have two concerns:

  • The practical benefit of η remains unclear. It does not reliably predict downstream gains, and the selection rule shows no clear advantage over DAPO at matched total compute.
  • Proposition 5 still has a logical gap. The signal of an individual prompt may rise and then fall as its success rate increases. However, different prompts can learn at different rates, so the claimed ordering of crossing times and the subsequent reversal do not necessarily follow from monotonic improvement alone.
AuthorsOpenReview ↗

Follow-up to Reviewer LHuq

Thank you for engaging with the revision. Both remaining points are fair, and we have fixed the first in the paper.

You are right about Proposition 5. Monotone progress of each prompt gives only part (i), that a prompt's signal rises until it crosses p=1/2p = 1/2 and falls after. The ordering of crossing times needs a second assumption, that training preserves the order of prompts, and with it part (ii) follows in one line (if yy has crossed and p0(x)≥p0(y)p_0(x) \ge p_0(y), then pt(x)≥pt(y)≥1/2p_t(x) \ge p_t(y) \ge 1/2). The set-level reversal needs more again. The revised Proposition 5 states it under an explicit idealization, a common shift in logit space, where two concentrated sets swap order exactly when (zS−zC)(zS+zC+2ϕ)<0(z_S - z_C)(z_S + z_C + 2\phi) < 0 changes sign. So a lead can be lost only if the control starts further below the mode than the selected set, which is the Countdown case, and never if it starts above, which is the GSM8K case. We also tested both assumptions on the one design that records every prompt's outcomes over time (App. F). Order is preserved at the level of starting-difficulty bins in all four segments of training, but a common rate is not. Easy-to-mid prompts climb by 0.6 to 1.0 in logit while near-mode prompts do not rise. The paper now says the proposition predicts where a reversal can and cannot occur, not when, and the proof is in App. A.

On practical benefit, we agree that η\eta does not predict a run's final gain and that the static rule does not beat DAPO at equal total compute. Its practical use is earlier and cheaper than either. From one sampling pass, before any RL compute is spent, it shows how much of a budget will carry a correctness gradient. That flags failures accuracy hides. Our overtrained warm-up had the highest held-out accuracy in its sweep and would have turned 4,800 generations into 312, and the base model's apparent learning signal was almost entirely format reward. When the pass is used to choose prompts, it reaches DAPO's accuracy at equal total generations with no change to the trainer and a cost known in advance.

Dear Authors,

Thank you for your detailed responses and the additional experiments. I greatly appreciate the effort you have put into addressing the feedback. I have carefully reviewed the manuscript, your rebuttal, and the other reviewers' comments. In my view, some limitations remain unaddressed. After considering your rebuttal and the subsequent discussion, I will maintain my original rating.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response to all reviewers

We thank the three reviewers for careful and specific reviews. They converge on three questions, and we have answered each with new derivations and experiments. A revised PDF is uploaded, and the changes are listed in a separate comment.

1. Does advantage mass track the gradient that actually updates the policy, in size and direction? (LHuq, znrf) We now prove (Prop. 3) that for a binary reward the exact per-prompt policy gradient is ∇p(x)=p(1−p)(μ+−μ−)\nabla p(x) = p(1-p)(\mu^+ - \mu^-), where μ±\mu^\pm are the mean score functions of successful and failed completions. The p(1−p)p(1-p) that η\eta aggregates is therefore the weight each prompt carries in the true gradient, not only a bound on an estimator's norm. RLOO's group gradient is unbiased for it. We then measured this on the checkpoints the study trains from (new Sec. 5.1, App. E). A group's gradient norm tracks its normalized mass at Spearman 0.55 to 0.68 in all six prompt sets. The score contrast ∥μ+−μ−∥\|\mu^+-\mu^-\| is flat in pp, with bin medians within 1.38x. And training-size minibatch updates from the high-η\eta set of each pair have higher cosine with an unbiased G=64G=64 estimate of the held-out gradient, by +0.056 to +0.068, with intervals excluding zero.

2. Is selecting prompts by p(1−p)p(1-p) new? No. The revision says so in the contributions, which now separate what was known from what is new. The rule is the instrument we use to move η\eta at fixed weights. We nevertheless compared it, charging every generation, with DAPO's dynamic sampling, a difficulty band, dropping silent prompts and online refilling (Table 11), and we redrew both the selected and the random sets in all four settings (Table 10). At equal total generations the rule is level with DAPO. Across the four settings, every one of three independently built selected sets beats every one of five random draws.

3. If η\eta does not predict the gain, what is it for? It is an accounting of the update, not a forecast of final accuracy, and the revision defines effective compute that way (Sec. 3.2). Early in a run it does what an accounting should. When we gave the random control enough steps to match the selected arm's effective compute at step 50, the gap in gain fell from +0.020 and +0.059 to +0.002 and +0.003, with intervals including zero. Over a whole run it does not equalize arms, because a selected set spends its signal first (Prop. 5). Our gradient-alignment measurements show the same course, with the advantage largest at step 0 and reversed by step 100.

We also restructured the paper around one claim, stated in the first paragraphs. In group-relative RL the gradient a fixed budget buys is set by the policy's outcome variance on its prompt set, it can be measured before training, and two routine decisions change once it is measured. Material that did not serve that claim moved to the appendix to make room for the new results within the 10-page limit.

AuthorsOpenReview ↗

Revision note

We have uploaded a further revision. Proposition 5 is restated with its assumptions explicit, its proof moved to App. A, and App. F tests those assumptions on per-prompt training records. verify_paper.py now checks 104 claims.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.