acceptodds
Under review as a conference paper at ICLR 2027

Shapley-Guided Rollout Selection for Reinforcement Learning with Verifiable Rewards

Abstract

Reinforcement learning with verifiable rewards improves language-model reasoning, but reducing training costs through response selection risks losing useful learning signals that reward statistics alone cannot identify. We aim to select a small subset of generated responses that preserves the complete group's learning direction under a fixed update budget. We introduce Shapley Selection Rollouts (SSL), which values each response by its average marginal contribution to recovering the group's direction across different response subsets. SSL measures this contribution using compact token-score representations as proxies for update directions, then passes the highest-valued responses to the existing policy optimizer without changing its objective. Across three language models and seven mathematics benchmarks, including GSM8K, MATH500, and AIME24/25, SSL improves mean benchmark accuracy by 0.49–0.62 percentage points over the strongest evaluated alternative when selecting 16 of 64 responses per prompt. By valuing responses in relation to one another, SSL provides a practical approach to preserving useful learning signals within a limited policy-update budget.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, reinforcement learning with verifiable rewards, rollout selection, Shapley values

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_hpLhOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies rollout down-sampling in GRPO-style RLVR: generate n=64 responses per prompt, select m=16 for the policy update. The key observation is that responses with similar rewards can induce reinforcing or canceling gradient directions, so selecting by individual reward statistics may lose useful signal. The proposed method, Shapley Selection Rollouts (SSL), values each rollout by its average marginal contribution to a coalition utility — the cosine alignment between a subset's advantage-weighted score aggregate and the full-group reference, computed in a d=128-dim token-score space. Top-m rollouts go to the unchanged GRPO objective. Exact Shapley for n≤16, Monte Carlo (K=512) otherwise. A paired normalization control separates "which rollouts" from "how advantages are normalized." Theorem 1 gives a progress guarantee under L-smoothness and bounded gradient discrepancy. Experiments cover three Qwen backbones, seven math benchmarks, TACO/LiveCodeBench, comparisons against PODS/CPPO/D3S-Sample/CBS*-quota (fixed pool) and D3S/CBS/ARRoL (matched budget), plus gradient-fidelity diagnostics and a conditional analysis on reward balance and directional conflict.

Critical strengths

Problem formulation and experimental controls. The gap the paper identifies — selectors value rollouts independently while their gradient contributions interact — is real. The paired normalization control (Section 3.6) cleanly isolates "which rollouts" from "how to weight them" by fixing the selected index set and varying only the advantage computation. Combined with independent parameter-gradient sketches (not the selection proxy), this gives the fidelity analysis credibility beyond downstream accuracy alone, and provides a template for evaluating future selectors.

Conditional analysis (Section 4.5). Decomposing groups by reward balance B_N and directional conflict C_N, then showing SSL's fidelity gain over PODS concentrates in the high-high region (Δ_fid=0.1091 vs. near-zero elsewhere), is more useful than the headline accuracy numbers. B_N and C_N are cheap to compute and could serve as a diagnostic for when joint selection is worth the cost. If the paper framed SSL as conditional on this diagnostic rather than a universal replacement, the contribution would be more defensible.

Critical weaknesses

Shapley necessity not demonstrated. The paper positions context-dependent joint valuation as the core innovation and uses Shapley values — O(2ⁿ) exact or K-permutation Monte Carlo — as the implementation. Yet no experiment tests whether a cheaper joint-valuation method on the same score proxy achieves similar results. Greedy forward selection (argmax_i v(S∪{i}), O(nm)) is the natural strawman; submodular or DPP variants are others. If these reach gradient-fidelity cosine within ~0.02 of SSL's 0.79 at a fraction of the cost, the Shapley machinery adds complexity without proportional benefit. The ablation space (Tables 3, 12, 13) varies only d and K — both internal to SSL — leaving the reader unable to distinguish "Shapley is the right tool" from "any joint-valuation method would do." Because the paper's methodological contribution IS the Shapley formulation, this gap undermines the central claim.

Gains not statistically significant and evidence chain incomplete. The 0.49–0.62 pp Mean improvement is within the reported five-seed SDs; approximate t-tests from Table 6 give p≈0.20/0.10/0.13 across the three backbones, none significant at 0.05. The short-response cohort (Table 1), which carries the most granular margins (+0.19–0.30 pp over D3S-Sample), withholds dispersion entirely. SSL also loses on GSM8K across all three backbones (Table 6) and on AIME pass@8 under two of three matched budgets (Table 2), with the Mean advantage carried by mid-difficulty benchmarks where per-benchmark SDs (0.6–0.9 pp) are comparable to the margins. Beyond significance: the claim that "reward statistics alone cannot identify" useful signals is contradicted by the paper's own Table 1, where the strongest baseline (D3S-Sample) is not a pure reward-statistic method, and SSL's margin over it is the smallest reported. Finally, the mechanism explanation — higher fidelity → better accuracy — is never tested directly: fidelity gains (Section 4.5) and accuracy gains are analyzed on different cohorts with no cross-reference. For a sub-1-pp method at 8% higher cost, this evidence chain is too thin.

Questions for authors

Q1: On the same d=128 proxy and frozen groups from Tables 11–12, what gradient-fidelity cosine does greedy forward selection (argmax_i v(S∪{i}) until |S|=m) achieve? If the gap to SSL's 0.79 is small, what justifies Shapley's overhead over a simple greedy scheme?

Q2: The 64→16 setting generates 64 rollouts but only updates on 16, so rollout generation dominates cost — especially for hard problems with long responses. The reported 8% cost difference between SSL and PODS/D3S-Sample is therefore driven by selection computation, not by training savings. Have you measured the generation-vs-update cost breakdown? If generation is the bottleneck, does selecting 16 of 64 actually reduce the dominant cost, or is the saving marginal relative to the full pipeline?

Q3: All three backbones are from the Qwen family. What is the rationale for this choice, and do you expect SSL to transfer to other families (Llama, Mistral)? More importantly: small models have low rollout success rates, meaning many zero-reward responses and high directional conflict — exactly the regime where Section 4.5 predicts SSL benefits most. With a larger model (e.g., 14B+) where success rates rise and the reward distribution shifts, would SSL's advantage shrink or disappear?

Q4: Section 4.5 shows fidelity gains concentrate in high-B_N, high-C_N groups. Do these same groups show the largest downstream accuracy gains for SSL vs. PODS? Without this link, the mechanism explanation remains correlational.

Q5: Experiments are limited to math and code, where rewards are binary or test-case-based and response structure is relatively uniform. How would SSL behave on QA/search tasks or verifiable agent trajectories, where reward distributions, response lengths, and the structure of "useful negative signal" differ substantially? Is there evidence that the score-proxy and coalition utility generalize beyond math/code?

6 comments
AuthorsOpenReview ↗

Response to Reviewer hpLh (1/4)

Thank you for taking the time to give such a detailed review and for your constructive suggestions. We address your questions below, focusing on the joint-selection baselines, the link between fidelity and accuracy, and transfer across models and tasks.

W1 / Q1. Does Shapley provide value beyond cheaper joint selection?

Response.{\color{green}\mathbf{Response.}} A response's training value depends on the responses selected alongside it. Adding a response changes the coalition's reward center and signed direction, including the contributions of existing members. Shapley valuation captures this dependence by averaging each response's marginal contribution across coalition contexts. Greedy evaluates additions along one construction path. Your suggested comparison tests the practical value of that averaging.

All proxy methods below use the same frozen groups, d=128d=128, and 64!→!1664\\!\to\\!16 quota. Best-pair initialization handles the zero-utility singleton tie explicitly.

SelectorGradient cosine: subset / full normalizationRelative error: subset normalizationFeature + selection time (ms/group)
PODS0.72 / 0.750.750.10
Greedy, index tie0.75 / 0.780.706.8
Greedy, best-pair initialization0.76 / 0.790.6710.9
OMP-mask0.755 / 0.7850.6815.1
SSL, K=512K=5120.79 / 0.820.63120.7
Paired comparisonCosine differenceGroup-bootstrap 95% CI
SSL − best-pair greedy+0.030[0.010, 0.050]

The same ordering appears downstream, including after charging SSL's overhead to a common budget:

MethodMean at 400 updates (%)Updates within 320 A100-GPU-hMean within that budget (%)
PODS41.91 ± 0.6141041.80 ± 0.60
Best-pair greedy42.38 ± 0.5740042.38 ± 0.57
SSL42.88 ± 0.5637042.56 ± 0.55

Greedy reaches its 400-update cap at 319.1 GPU-hours. SSL improves the mean by +0.50 pp at fixed updates and +0.18 pp within the common budget. The paired interval supports the fidelity advantage, and the downstream means show the same ordering after accounting for cost. This gives empirical support for averaging contributions across coalition contexts.


W2a. Are the small gains significant, and how should the negative results be interpreted?

Response.{\color{green}\mathbf{Response.}} The matched runs allow a direct test of your concern. A paired test uses the covariance between methods, which is absent from a calculation based only on their marginal SDs. The observed differences for Table 6 are:

BackboneSSL−CBS*-quota, seeds 0–4 (pp)Mean differencePaired 95% tt CIHolm-adjusted pp
Qwen2.5-3B0.42, 0.62, 0.16, 0.36, 0.89+0.49[0.146, 0.834]0.0292
Qwen2.5-Math-7B0.49, 0.72, 0.29, 0.52, 1.08+0.62[0.249, 0.991]0.0292
Qwen3-4B0.85, 0.92, 0.23, 0.32, 0.78+0.62[0.222, 1.018]0.0292

All five differences are positive for each backbone. The paired-tt intervals exclude zero under an approximately normal-difference assumption. Holm-adjusted pp values are 0.0292 for all three comparisons. The exact two-sided sign-flip test gives p=0.0625p=0.0625, its smallest attainable value with five pairs. These inferential results apply to Table 6's aggregate score. Table 1 provides descriptive five-seed means for the short-response tasks.

For your GSM8K observation, Table 6 shows a +0.35 pp gain on Qwen2.5-Math-7B and small decreases on the other two backbones. The table below also includes all three AIME budget comparisons.

Original resultSSLComparatorDifference (pp)
Qwen2.5-3B, GSM8K80.85CBS*-quota: 80.95−0.10
Qwen2.5-Math-7B, GSM8K92.05CBS*-quota: 91.70+0.35
Qwen3-4B, GSM8K89.50CBS*-quota: 89.65−0.15
AIME, equal generated tokens59.20ARRoL: 59.35−0.15
AIME, equal GPU-hours58.95ARRoL: 59.10−0.15
AIME, equal update tokens59.35ARRoL: 59.05+0.30

The seven-benchmark aggregate improves consistently across paired runs. The task-level results specify where that gain occurs, including the small GSM8K and AIME decreases shown above.

AuthorsOpenReview ↗

Response to Reviewer hpLh (2/4)

W2b. Does D3S-Sample contradict the limitation of reward-statistic selection?

Response.{\color{green}\mathbf{Response.}} You correctly point out that full D3S includes cross-group allocation, entropy-based token selection, and a sampling schedule [1]. Table 2 evaluates that native system. D3S-Sample is the fixed-quota, within-prompt selection adaptation described in Appendix B.2.

Under this controlled condition, full-group normalization gives

=\frac{\operatorname{Var}_{i\in S}(r_i)}{(\sigma_N+\varepsilon)^2}.$$ The maximizing subsets therefore coincide with reward-variance maximizers up to ties in this specific setting. The following control isolates the additional information used by SSL by fixing its selected positive/negative reward counts: | Selector | Gradient cosine: subset normalization | Gradient cosine: full normalization | |---|---:|---:| | PODS / sign balance | 0.72 | 0.75 | | Random within SSL's reward composition | 0.66 | 0.72 | | SSL | **0.79** | **0.82** | At identical reward composition, SSL achieves higher fidelity by choosing responses whose directions better preserve the full-group signal. This is the contribution of joint directional valuation measured by the control. [1] Learning More with Less: A Dynamic Dual-Level Down-Sampling Framework for Efficient Policy Optimization. --- > #### **W2c / Q4. Do the same high-balance/high-conflict prompts show larger accuracy gains?** ${\color{green}\mathbf{Response.}}$ Yes, in the held-out cohort evaluated for this question. A common reference checkpoint defines the balance/conflict strata before method-specific training. The final SSL and PODS models are evaluated on those same 512 prompts with 32 responses per prompt and five training seeds. All strata, including the 16 degenerate prompts, are shown below. | Reference-defined region | Prompts | SSL−PODS fidelity difference (95% CI) | SSL−PODS final accuracy difference, pp (95% CI) | |---|---:|---:|---:| | $B>0.75,\\ C>0.65$ | 148 | **+0.1091 [0.085, 0.133]** | **+2.14 [1.25, 3.03]** | | $B>0.75,\\ C<0.40$ | 28 | +0.0290 [0.002, 0.056] | +0.60 [−0.40, 1.60] | | $B<0.40,\\ C>0.65$ | 6 | +0.0277 [−0.030, 0.086] | +0.40 [−2.10, 2.90] | | $B<0.40,\\ C<0.35$ | 43 | −0.0028 [−0.025, 0.019] | −0.10 [−0.70, 0.50] | | Other non-degenerate regions | 271 | +0.0509 [0.035, 0.067] | +0.55 [0.10, 1.00] | | Zero-reference/reward-degenerate | 16 | N/A | 0.00 | | All eligible regions | 496 | +0.0621 [0.048, 0.076] | +0.97 [0.60, 1.34] | | Prompt-level association | Estimate (95% CI) | |---|---:| | Partial Spearman between fidelity and accuracy differences, controlling benchmark and initial success | **0.26 [0.12, 0.39]** | This is a separate held-out cohort from Table 15's 2,483 eligible groups. A cloned-state intervention also tests actual optimizer behavior: | Method | AdamW displacement cosine to full GRPO | Frozen-objective improvement ratio after norm matching (95% CI) | |---|---:|---:| | PODS | 0.75 | 0.78 [0.73, 0.83] | | Best-pair greedy | 0.78 | 0.84 [0.79, 0.89] | | SSL | **0.83** | **0.90 [0.86, 0.94]** | The same high-balance/high-conflict prompts show the largest accuracy gain, as you asked. The partial correlation quantifies the association between fidelity and accuracy gains after controlling for benchmark and initial success. The cloned-state experiment provides complementary evidence that SSL preserves local improvement under actual AdamW updates.
AuthorsOpenReview ↗

Response to Reviewer hpLh (3/4)

W2d / Q2. Does generation dominate, and how much does selecting 16 of 64 actually save?

Response.{\color{green}\mathbf{Response.}} You are right that all methods still generate 64 responses. Savings relative to full GRPO come from the policy update. The measured breakdown for the 3,072-token setting on eight A100s is:

MethodGenerationVerificationFeatures + selectionUpdate + communicationOtherTotal
Full GRPO, 64→642651402606545
PODS, 64→16265142.064.06351
Best-pair greedy, 64→16270143.066.06359
SSL, 64→16294144.570.56389

All entries are seconds per step. Generation accounts for 75.6% of SSL's total. Reducing the update workload still lowers end-to-end step time by 28.6% relative to full GRPO. SSL costs 10.8% more than PODS in this setting. That difference includes generation and update time as well as selection, because the independently trained SSL policy produces longer responses on average.

MethodMean response lengthGenerated tokens/updateUpdate tokens/updateMean within 320 A100-GPU-h (%)
Full GRPO1,6501,689,6001,689,60041.15 ± 0.58
PODS1,6501,689,600407,04041.80 ± 0.60
Best-pair greedy1,6801,720,320422,40042.38 ± 0.57
SSL1,8381,882,112440,32042.56 ± 0.55

Token totals cover 16 prompts per update. Greedy stops at 400 updates, consuming 319.1 GPU-hours within the 320-hour cap. In Table 1's short-response setting, SSL/PODS cost 5.2/4.8 GPU-hours, an 8.3% total increase. Selection accounts for 1.92% of SSL's time. The budget comparison includes generation, selection, and updates, and SSL has the highest mean accuracy under that cap.


Q3. Does the result transfer beyond Qwen, and what changes at 14B?

Response.{\color{green}\mathbf{Response.}} The original checkpoints overlap the evaluation settings of PODS, D3S/VIP, and CBS/ARRoL (Appendix B.4). Your family-transfer question is tested with Llama-3.1-8B-Instruct, using the extended mathematics protocol, 400 updates, and five seeds:

MethodGradient cosineMath Mean ± SD (%)Training GPU-hours
PODS0.71 ± 0.1544.10 ± 0.58310
Best-pair greedy0.75 ± 0.1444.32 ± 0.57314
SSL0.78 ± 0.1344.61 ± 0.56322
Llama paired comparisonMean difference (pp)Reported paired 95% CI
SSL − PODS+0.51approximately [0.08, 0.94]

For your scale question, reward balance is highest at an intermediate success rate. With binary rewards, BN=4qN(1−qN)B_N=4q_N(1-q_N) peaks at qN=0.5q_N=0.5 and is small near both zero and one. Conflict additionally depends on the signed directions. The frozen comparison on common prompts gives:

Qwen2.5-Instruct modelMean success rateHigh-BB/high-CC group shareProxy–gradient correlationPODS / SSL cosine
3B0.5629%0.680.72 / 0.79
14B0.7814%0.620.74 / 0.765

At 14B, the high-balance/high-conflict share falls from 29% to 14%, and SSL's fidelity advantage narrows from 0.07 to 0.025. The frozen diagnostic therefore supports your proposed dependence on the group's balance/conflict regime. The Llama experiment provides the complementary downstream training evidence across model families.

Reviewer_pp3vOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes rollout downsampling method which can be used for RLVR named Shapley Selection Rollouts. The authors claim that existing rollout downsampling methods usually select rollouts using reward statistics. However, proposed method, SSL tries to select a subset that best preserves update direction of full group. Based on score feature and calculated coalition utility, SSL selects the rollouts with highest Shapley value and use it for a gradient update. As a result, this method obtains better performance compared to the existing baselines.

Critical strengths

The problem definition and idea seem reasonable. The core idea of this method is to select which rollouts to be used based on the joint contribution, not on independent reward statistics, which make a clear difference from previous methods. Recomputation of advantage for each coalition is also clear and acceptable direction.

This paper contains wide range of experiments which support that obtained gain comes from a subset selection itself.

Critical weaknesses

I’m not sure that why the score proxy can be used as a proxy of gradient update direction. Even though this paper assume bridge function to connect score and gradient, the experiments does not calibrate or estimate this value.

In addition, overall performance gain seems to be within a noise level. Most of performance gain is within or similar level with a standard deviation compared to the strongest baseline. Since this method requires additional training cost compared to the other baselines, this small gap makes the method less attractive. Statistical significance test might strengthen the claim on effectiveness of the method.

Questions for authors

Can you suggest more evidence or clarification on the why score proxy should be used as a proxy of gradient update direction?

3 comments
AuthorsOpenReview ↗

Response to Reviewer pp3v

Thank you for your time and for the helpful comments on proxy calibration and the size of the gains. Below, we address both points with gradient measurements, paired tests, and comparisons under a common training budget.

W1 / Q1. What supports using the score representation as a gradient-direction proxy?

Response.{\color{green}\mathbf{Response.}} The score vector summarizes a response's token log-probability profile. It retains differences between responses that receive the same reward. For binary rewards and a coalition containing both classes, its aggregate satisfies

\left(\bar z_{S,+}-\bar z_{S,-}\right),$$ where $q_S$ is the positive-reward fraction. The direction depends on the positive- and negative-response centroids, even when reward counts are fixed. Your question about how this feature direction tracks parameter gradients is addressed by the following calibration. On the frozen Qwen3-4B checkpoint, correlations across coalitions are: | Feature | Within-group Pearson $r$ (prompt-cluster 95% CI) | Within-group Spearman $\rho$ (95% CI) | |---|---:|---:| | Positional log-probability proxy | **0.68 [0.59, 0.75]** | **0.66 [0.57, 0.74]** | | Mean log-probability + length | 0.24 [0.13, 0.34] | 0.21 [0.10, 0.32] | | Shuffled token positions | 0.08 [−0.03, 0.19] | 0.06 [−0.05, 0.17] | Among coalitions with proxy cosine $\geq0.8$, the same controls show: | Feature | Gradient cosine P05 | Negative-gradient-cosine fraction | |---|---:|---:| | Positional proxy | **0.39** | **2.3%** | | Mean log-probability + length | −0.12 | 18% | | Shuffled token positions | −0.28 | 46% | The affine bridge is fitted and validated on separate prompt sets before testing: | Evaluation space | Held-out groups | Relative residual median / P90 | Original / mapped cosine correlation | |---|---:|---:|---:| | Independent 2,048-d gradient sketch | 128 | **0.18 / 0.30** | 0.68 / 0.75 | | Full trainable-parameter audit | 8 | 0.24 / 0.39 | 0.67 / 0.73 | The fitted map is used only for this calibration diagnostic. SSL selects responses directly from the original score features. The held-out residuals and eight-group full-parameter audit quantify the bridge at this checkpoint. The theoretical guarantee uses the bridge conditions in Proposition 4, with scaled isometry additionally required for Appendix D.5's ideal angle result. Your question also concerns actual updates. With identical cloned AdamW states, the local experiment gives: | Method | Displacement cosine to full GRPO | Norm-matched frozen-objective improvement / full GRPO (95% CI) | |---|---:|---:| | PODS | 0.75 | 0.78 [0.73, 0.83] | | Best-pair greedy | 0.78 | 0.84 [0.79, 0.89] | | SSL | **0.83** | **0.90 [0.86, 0.94]** | The calibration and AdamW experiments agree: the positional proxy retains directional information beyond reward and length, and SSL uses it to preserve more of the full update's local improvement. --- > #### **W2. Are the gains within noise, and do they justify the additional cost?** ${\color{green}\mathbf{Response.}}$ Your question calls for both a paired comparison and a full cost comparison. Matched runs share initialization and prompt order, so the relevant uncertainty is the variation in their paired differences. For the five training seeds in Table 6: | Backbone | SSL−CBS*-quota (pp) | Paired 95% $t$ CI | Holm-adjusted $p$ | Positive paired differences | |---|---:|---:|---:|---:| | Qwen2.5-3B | +0.49 | [0.146, 0.834] | 0.0292 | 5/5 | | Qwen2.5-Math-7B | +0.62 | [0.249, 0.991] | 0.0292 | 5/5 | | Qwen3-4B | +0.62 | [0.222, 1.018] | 0.0292 | 5/5 | All five differences are positive for each backbone. The paired-$t$ intervals exclude zero under the approximately normal-difference assumption, with Holm-adjusted $p=0.0292$. The exact two-sided sign-flip test gives $p=0.0625$, the smallest attainable value with five pairs. Charging all training overhead to the same budget gives: | Qwen3-4B method | Updates within 320 A100-GPU-h | Mean within that budget (%) | SSL mean advantage (pp) | |---|---:|---:|---:| | PODS | 410 | 41.80 ± 0.60 | +0.76 | | Best-pair greedy | 400 | 42.38 ± 0.57 | +0.18 | | ARRoL | Native budget protocol | 42.38 ± 0.57 | +0.18 | | SSL | 370 | **42.56 ± 0.55** | — | Greedy reaches its 400-update cap at 319.1 GPU-hours. SSL retains the highest mean within the budget after charging all training costs. In the short-response setting, SSL and PODS cost 5.2 and 4.8 GPU-hours (Table 1), an 8.3% total increase. The selector itself accounts for 1.92% of SSL's time. The budget table reports mean advantages, and the paired inference above quantifies the comparison with CBS*-quota.

Thank you for the detailed response and the additional experiments.

The new experiments on the score proxy largely address my concern there.

I am less convinced about the significance of the performance gains. The new paired test compares SSL only with CBS*-quota under the fixed-update protocol, which is not matched for compute: within the matched GPU-hour budget, SSL completes only 370 updates. In the matched GPU-hour setting, the strongest baseline is ARRoL, and there the difference is only +0.18 pp, which is small next to a standard deviation of about 0.55. ARRoL also beats SSL on AIME. My original concern was about this compute-matched comparison, and no significance test is given for it. In addition, with only five pairs the exact sign-flip test cannot go below p = 0.0625, so the significance claim depends on assuming that the five differences are normally distributed.

The paired analysis also assumes that runs sharing a seed are strongly correlated. But as Appendix B.2 notes, matched runs share only initialization and prompt order, and their rollouts diverge during training. In my experience, even runs with identical seeds often do not reproduce the same results.

A paired test of SSL against ARRoL under the matched GPU-hour budget, along with per-seed scores for both methods, would help show whether the gain really exceeds seed-level variance. For now, I will keep my current score.

AuthorsOpenReview ↗

Response to your follow-up: SSL versus ARRoL at matched compute

Thank you for your careful follow-up. Here is the direct compute-matched comparison you requested.

1. SSL versus ARRoL at matched compute

Response.{\color{green}\mathbf{Response.}} Both methods use Table 2's 320 A100-GPU-hour budget, including training overhead. SSL is evaluated at 370 updates. Mean is the seven-benchmark average, and AIME is average AIME24/25 pass@8. Scores are percentages and differences are percentage points (pp).

SeedSSL MeanARRoL MeanΔ MeanSSL AIMEARRoL AIMEΔ AIME
042.0341.98+0.0556.4556.94−0.49
142.4642.24+0.2259.0560.10−1.05
242.2042.01+0.1957.9057.10+0.80
342.6842.30+0.3859.0059.30−0.30
443.4343.37+0.0662.3562.06+0.29
Mean ± SD42.56 ± 0.5542.38 ± 0.57+0.180 ± 0.13558.95 ± 2.1859.10 ± 2.15−0.150 ± 0.715
Paired statistic: SSL − ARRoLMeanAIME pass@8
Independent matched pairs55
SD of differences (pp)0.1350.715
Observed Pearson rr0.9720.945
Paired-t 95% CI (pp)[0.012, 0.348][−1.038, 0.738]
Two-sided paired-t pp0.04080.6634
Two-sided exact sign-flip pp0.06250.6875
SSL wins / ties / losses5 / 0 / 02 / 0 / 3

SSL improves the seven-benchmark mean in all five matched pairs, averaging +0.18 pp at equal compute.


2. Pairing and statistical evidence

Response.{\color{green}\mathbf{Response.}} Appendix B.2 matches initialization and prompt order. The correlation is measured directly from these five pairs: r=0.972r=0.972. Rollout divergence is reflected in the observed paired differences. Their SD is 0.135 pp, giving a standard error of 0.060 pp for the mean gain. This is the uncertainty relevant to the paired comparison.

The paired-t test gives p=0.0408p=0.0408 and a positive 95% CI of [0.012, 0.348] pp, assuming normally distributed paired differences. The exact sign-flip test gives p=0.0625p=0.0625, the smallest attainable two-sided value with five pairs.


3. AIME and overall performance

Response.{\color{green}\mathbf{Response.}} As you noted, ARRoL leads on AIME by 0.15 pp, with outcomes varying across seeds. The seven-benchmark average incorporates this result and still improves in every matched pair.


We hope these results address your concerns. Thank you again for your time!

Reviewer_Baz7OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes to use shapley value to select subset of trajectories for GRPO calculation. More specifically, they define the game utility of the group being the cosine between a subset's advantage weigthed feature aggregation and teh same agg. of the full group. They also show that single step progress can be guaranteed even when the subgroup gradient is different from the full group. Experiments on common math benchmarks show improvements of the proposed method.

Critical strengths
  • the problem is relevant
  • using shapley value for GRPO group selection and also the consine metric is novel
Critical weaknesses
  • it doesn't make sense that the full group GRPO performs worst than the subgroup approximation (the proposed method) on every backbone (table 6)
  • The paper's own App. D.6 builds two policies with identical proxy observations in which the SSL-selected subset's gradient has the opposite sign to the full-group gradient. The bridge result (Prop. 4) needs H = BZ^⊤ + b1^⊤ + E with small ‖EC‖. In words, billion-parameter gradients must be roughly linear in 128 pooled log-probabilities. \tau is never estimated. This doesn't make sense in practice. It reads to me that the theory doesn't apply to the algo proposed.
  • The experiment section I think it's mostly generated via LLM directly. It lacks in depth results analysis but only plain "number dropping".
  • Figure 1 is overly aclutter and hard to extract useful information.
Questions for authors
  1. Why should positional log-probability profiles track update directions? Please report the correlation between proxy-space and gradient-sketch cosines across coalitions, and an estimate of \tau.
  2. Why does full group GRPO underperform subgropu GRPO?
  3. How do greedy selection on v(S), OMP, gradient-sketch selection and a sign-balancing heuristic perform at equal cost?
  4. Please provide per-seed results, paired tests and CIs for all tables, and resolve the GSM8K discrepancy.
5 comments
AuthorsOpenReview ↗

Response to Reviewer Baz7 (1/4)

Thank you for the time you spent reviewing our paper and for your thoughtful comments. We address your questions below, with calibration results, controlled comparisons, and a clearer explanation of the method.

Accuracy differences are in percentage points (pp). Training results are mean ± SD over five seeds unless stated otherwise.

W1 / Q2. Why can a selected subset outperform full-group GRPO?

Response.{\color{green}\mathbf{Response.}} The full-group gradient describes the current empirical update. Theorem 1 bounds one-step progress on a frozen surrogate under the stated gradient-discrepancy conditions. Final test accuracy depends on the full training trajectory and generalization, so a subset can achieve higher accuracy while approximating full-group updates. Section 4.1 of PODS [1] also reports higher final performance from downsampling in some settings.

To address your question, we compare both fixed updates and a common training-time budget. The Qwen3-4B accuracy values from Tables 6 and 2 and the measured costs are:

MethodMean at 400 updates (%)Cost for 400 updates (A100-GPU-h)Mean at 320 A100-GPU-h (%)
Full GRPO, 64→6441.29 ± 0.5948441.15 ± 0.58
PODS, 64→1641.91 ± 0.6131241.80 ± 0.60
SSL, 64→1642.88 ± 0.5634642.56 ± 0.55

A cloned-state AdamW experiment measures local progress. Each branch starts from identical parameters and optimizer state. Improvements are normalized by the full-group branch.

MethodActual displacement cosine to full GRPOFrozen-objective improvement ratio, common LR (95% CI)Ratio after displacement-norm matching (95% CI)
Full GRPO1.001.001.00
PODS0.750.76 [0.71, 0.81]0.78 [0.73, 0.83]
SSL0.830.88 [0.84, 0.92]0.90 [0.86, 0.94]

SSL recovers more of the full group's local improvement than PODS, including after displacement-norm matching. The final-accuracy results show that this closer local approximation is compatible with a better training outcome. Together, the experiments explain why the observed accuracy ordering is consistent with Theorem 1.

[1] Not All Rollouts are Useful: Down-Sampling Rollouts in LLM Reinforcement Learning.

AuthorsOpenReview ↗

Response to Reviewer Baz7 (2/4)

W2 / Q1. Why should pooled log-probabilities track gradients, and does Appendix D.6 invalidate the theory?

Response.{\color{green}\mathbf{Response.}} The key issue you raise is whether the proxy tracks parameter gradients. Appendix D.6 shows why a calibration condition is necessary. Proposition 4 states the bridge assumption under which the proxy supports the progress analysis. The following experiments measure how closely it holds.

The bridge is a local relation among a frozen group's rollout gradients, shared across that group's coalitions. We fit a ridge map on 64 calibration prompts and choose its regularization on 32 separate prompts. Evaluation uses 128 held-out groups. For the gradient sketch JJ, the measured residual is

\widehat\tau_J=\left\\|\left(JH-\widehat B_J Z^\top-\widehat b_J\mathbf1^\top\right)C\right\\|_{\mathrm{op}}.

Checkpoint / evaluation spaceTest groupsResidual median / P90Relative residual median / P90
Step 200 / 2,048-d gradient sketch1280.25 / 0.410.22 / 0.35
Step 400 / 2,048-d gradient sketch1280.21 / 0.360.18 / 0.30
Step 400 / full trainable-parameter audit80.29 / 0.460.24 / 0.39

Relative residual divides by the centered gradient operator norm in the corresponding space. These estimates characterize calibration on the held-out groups at the measured checkpoints. The eight-group audit also checks the residual in the full trainable-parameter space. Separate fitting and evaluation prompts test whether the fitted map transfers to new groups.

Across sampled coalitions of sizes 8, 16, and 32, within-group proxy/gradient cosine correlations are:

Checkpoint / featurePearson rr (95% CI)Spearman ρ\rho (95% CI)
Step 0 / positional proxy0.52 [0.41, 0.61]0.49 [0.38, 0.59]
Step 200 / positional proxy0.61 [0.52, 0.69]0.59 [0.49, 0.67]
Step 400 / positional proxy0.68 [0.59, 0.75]0.66 [0.57, 0.74]
Step 400 / mean log-probability + length0.24 [0.13, 0.34]0.21 [0.10, 0.32]
Step 400 / shuffled token positions0.08 [−0.03, 0.19]0.06 [−0.05, 0.17]

Intervals use prompt-cluster bootstrap. For coalitions with proxy cosine ≥0.8\geq0.8 at step 400, the gradient-direction tails are:

FeatureGradient cosine P05Negative-gradient-cosine fraction
Positional proxy0.392.3%
Mean log-probability + length−0.1218%
Shuffled token positions−0.2846%

The positional proxy has stronger correlation and fewer reversed directions than either control. This supports the value of token-position information beyond average confidence and length. The 2.3% reversal rate also quantifies the proxy's tail behavior. Proposition 4 retains its stated bridge conditions, and the ideal angle guarantee in Appendix D.5 additionally requires scaled isometry. Actual AdamW behavior is measured separately in the cloned-state experiment above.


W3. The experiments need analysis beyond reporting numbers.

Response.{\color{green}\mathbf{Response.}} Three controlled comparisons address your request for deeper analysis. The paper's AI-use statement describes assistance with literature, formula checking, presentation, and language. The experiments below examine what SSL preserves and when that information improves training:

Controlled comparisonMeasurementResult
Same positive/negative reward counts as SSLGradient cosine: matched random / SSL0.66 / 0.79
Same optimizer state and matched displacement normLocal improvement ratio: PODS / SSL0.78 / 0.90
Same held-out prompts, high balance and conflictSSL−PODS final accuracy, 95% CI+2.14 pp [1.25, 3.03]
Same held-out prompts, low balance and conflictSSL−PODS final accuracy, 95% CI−0.10 pp [−0.70, 0.50]

The reward-matched control isolates the value of selecting responses within the same reward composition. Norm matching shows that SSL's local advantage persists at a common update magnitude. The same-prompt analysis then links larger fidelity gains to larger accuracy gains in the high-balance/high-conflict region. This combination of controlled local evidence and downstream association explains the conditions in which joint selection is most useful.

AuthorsOpenReview ↗

Response to Reviewer Baz7 (3/4)

W4. Figure 1 is difficult to follow.

Response.{\color{green}\mathbf{Response.}} The essential flow in Figure 1 is:

Generate 64 responses → measure each response's contribution across coalitions → rank by mean marginal contribution → update on the top 16.

For each coalition, rewards are re-centered and its signed score aggregate is compared with the fixed full-group reference. The change in this alignment defines a response's marginal contribution. Selection uses cached, 128-dimensional pooled token log-probabilities, so it requires no per-response backward pass. The top-ranked responses enter the existing GRPO loss.


Q3. How do greedy, OMP, gradient-sketch selection, and sign balancing compare at equal cost?

Response.{\color{green}\mathbf{Response.}} Your proposed alternatives are evaluated on the same 122 nonzero-reference groups, with n=64n=64 and m=16m=16. Proxy-based methods share the same d=128d=128 representation. Singleton utilities are zero, so both index-tie greedy and a stronger best-pair initialization are included.

SelectorGradient cosine, subset / full normalizationRelative error, subset normalizationFeature/backward + selector time (ms/group)
PODS0.72 / 0.750.750.10
Sign balance0.72 / 0.750.750.02
SSL reward-count-matched random0.66 / 0.720.840.02
Greedy, index tie0.75 / 0.780.706.8
Greedy, best-pair initialization0.76 / 0.790.6710.9
OMP-mask0.755 / 0.7850.6815.1
Gradient-sketch greedy0.82 / 0.830.55496.0
SSL, K=512K=5120.79 / 0.820.63120.7

For reward-count-matched random, 0.02 ms covers only the random draw after SSL supplies the reward counts. This control isolates response identity while keeping reward composition fixed.

Paired comparisonCosine differenceGroup-bootstrap 95% CI
SSL − best-pair greedy+0.030[0.010, 0.050]

OMP-mask uses the selected indices with the existing GRPO weighting. This isolates subset selection from OMP coefficient reweighting in the related GRAD-MATCH formulation [1]. The gradient-sketch baseline achieves the highest fidelity, with its backward computation included in the reported cost.

For the strongest inexpensive proxy alternative, the end-to-end budget comparison is:

MethodMean at 400 updates (%)Updates within 320 A100-GPU-hMean within that budget (%)
PODS41.91 ± 0.6141041.80 ± 0.60
Greedy, best-pair initialization42.38 ± 0.5740042.38 ± 0.57
SSL42.88 ± 0.5637042.56 ± 0.55

Greedy reaches its 400-update cap at 319.1 GPU-hours. SSL's mean advantage is +0.50 pp at fixed updates and +0.18 pp within the common budget. The paired interval quantifies the fidelity advantage, while the downstream comparison reports five-seed means and SDs. Both comparisons favor SSL over the strongest tested inexpensive proxy selector.

[1] GRAD-MATCH: Gradient Matching based Data Subset Selection for Efficient Deep Model Training.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.