Matched Compute Is Not Matched Signal: Where the Gradient Goes in Group-Relative Reinforcement Learning

Reinforcement learning from verifiable rewards is usually preceded by a supervised warm-up, and runs are compared at matched compute. But group-relative methods such as GRPO learn only from prompts whose sampled answers disagree: a group of rollouts that all earn the same reward contributes exactly zero gradient, however much it cost to generate. Two runs with the same budget can therefore receive very different amounts of learning signal. Prior work uses this to filter prompts during training; we make it exact and extend it to policies and prompt sets. We derive the gradient signal a group carries under GRPO, Dr.GRPO and RLOO and average it over the prompts. The result is a signal efficiency $\eta =c_G[a(1-a)-{\mathrm{Var}}_x(p)]$, where $a$ is mean accuracy, $p$ is a prompt's success probability and $c_G$ depends only on the group size. A more accurate policy can therefore carry less signal. Across 32 runs given identical budgets, the generations that actually carried a gradient differed by a factor of $8.8$. A warm-up trained on the problems its reinforcement-learning phase will use silences them: after $27$ epochs, solution distillation solves $98.6\%$ of them and $\eta$ falls from $0.600$ to $0.065$. The same accounting explains why a base model barely improves: its gradient is real, but almost all of it comes from the format reward rather than from being right. Choosing prompts by $p(1-p)$, from a single sampling pass before training, beats a random set of the same size and budget on all $20$ runs across four settings, a search task and GSM8K at 3B to 14B, by $2.2$ to $7.8$ points of held-out accuracy. The lead survives at twice the training horizon, though its signal advantage fades as the chosen prompts are solved, and reverses on every seed in the one setting where the random control climbs far enough toward $p=1/2$. Two plausible variants of the rule each fail in some setting, by overshooting an optimum the accounting locates. What $\eta$ does not do is rank warm-ups or predict how much a run will gain: it prices prompts, not checkpoints. Runs should therefore be compared by the generations that carry a gradient, not the generations spent. Code: https://anonymous.4open.science/r/signal-efficiency-rl-D5C1.