- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies rollout down-sampling in GRPO-style RLVR: generate n=64 responses per prompt, select m=16 for the policy update. The key observation is that responses with similar rewards can induce reinforcing or canceling gradient directions, so selecting by individual reward statistics may lose useful signal. The proposed method, Shapley Selection Rollouts (SSL), values each rollout by its average marginal contribution to a coalition utility — the cosine alignment between a subset's advantage-weighted score aggregate and the full-group reference, computed in a d=128-dim token-score space. Top-m rollouts go to the unchanged GRPO objective. Exact Shapley for n≤16, Monte Carlo (K=512) otherwise. A paired normalization control separates "which rollouts" from "how advantages are normalized." Theorem 1 gives a progress guarantee under L-smoothness and bounded gradient discrepancy. Experiments cover three Qwen backbones, seven math benchmarks, TACO/LiveCodeBench, comparisons against PODS/CPPO/D3S-Sample/CBS*-quota (fixed pool) and D3S/CBS/ARRoL (matched budget), plus gradient-fidelity diagnostics and a conditional analysis on reward balance and directional conflict.
Critical strengths
Problem formulation and experimental controls. The gap the paper identifies — selectors value rollouts independently while their gradient contributions interact — is real. The paired normalization control (Section 3.6) cleanly isolates "which rollouts" from "how to weight them" by fixing the selected index set and varying only the advantage computation. Combined with independent parameter-gradient sketches (not the selection proxy), this gives the fidelity analysis credibility beyond downstream accuracy alone, and provides a template for evaluating future selectors.
Conditional analysis (Section 4.5). Decomposing groups by reward balance B_N and directional conflict C_N, then showing SSL's fidelity gain over PODS concentrates in the high-high region (Δ_fid=0.1091 vs. near-zero elsewhere), is more useful than the headline accuracy numbers. B_N and C_N are cheap to compute and could serve as a diagnostic for when joint selection is worth the cost. If the paper framed SSL as conditional on this diagnostic rather than a universal replacement, the contribution would be more defensible.
Critical weaknesses
Shapley necessity not demonstrated. The paper positions context-dependent joint valuation as the core innovation and uses Shapley values — O(2ⁿ) exact or K-permutation Monte Carlo — as the implementation. Yet no experiment tests whether a cheaper joint-valuation method on the same score proxy achieves similar results. Greedy forward selection (argmax_i v(S∪{i}), O(nm)) is the natural strawman; submodular or DPP variants are others. If these reach gradient-fidelity cosine within ~0.02 of SSL's 0.79 at a fraction of the cost, the Shapley machinery adds complexity without proportional benefit. The ablation space (Tables 3, 12, 13) varies only d and K — both internal to SSL — leaving the reader unable to distinguish "Shapley is the right tool" from "any joint-valuation method would do." Because the paper's methodological contribution IS the Shapley formulation, this gap undermines the central claim.
Gains not statistically significant and evidence chain incomplete. The 0.49–0.62 pp Mean improvement is within the reported five-seed SDs; approximate t-tests from Table 6 give p≈0.20/0.10/0.13 across the three backbones, none significant at 0.05. The short-response cohort (Table 1), which carries the most granular margins (+0.19–0.30 pp over D3S-Sample), withholds dispersion entirely. SSL also loses on GSM8K across all three backbones (Table 6) and on AIME pass@8 under two of three matched budgets (Table 2), with the Mean advantage carried by mid-difficulty benchmarks where per-benchmark SDs (0.6–0.9 pp) are comparable to the margins. Beyond significance: the claim that "reward statistics alone cannot identify" useful signals is contradicted by the paper's own Table 1, where the strongest baseline (D3S-Sample) is not a pure reward-statistic method, and SSL's margin over it is the smallest reported. Finally, the mechanism explanation — higher fidelity → better accuracy — is never tested directly: fidelity gains (Section 4.5) and accuracy gains are analyzed on different cohorts with no cross-reference. For a sub-1-pp method at 8% higher cost, this evidence chain is too thin.
Questions for authors
Q1: On the same d=128 proxy and frozen groups from Tables 11–12, what gradient-fidelity cosine does greedy forward selection (argmax_i v(S∪{i}) until |S|=m) achieve? If the gap to SSL's 0.79 is small, what justifies Shapley's overhead over a simple greedy scheme?
Q2: The 64→16 setting generates 64 rollouts but only updates on 16, so rollout generation dominates cost — especially for hard problems with long responses. The reported 8% cost difference between SSL and PODS/D3S-Sample is therefore driven by selection computation, not by training savings. Have you measured the generation-vs-update cost breakdown? If generation is the bottleneck, does selecting 16 of 64 actually reduce the dominant cost, or is the saving marginal relative to the full pipeline?
Q3: All three backbones are from the Qwen family. What is the rationale for this choice, and do you expect SSL to transfer to other families (Llama, Mistral)? More importantly: small models have low rollout success rates, meaning many zero-reward responses and high directional conflict — exactly the regime where Section 4.5 predicts SSL benefits most. With a larger model (e.g., 14B+) where success rates rise and the reward distribution shifts, would SSL's advantage shrink or disappear?
Q4: Section 4.5 shows fidelity gains concentrate in high-B_N, high-C_N groups. Do these same groups show the largest downstream accuracy gains for SSL vs. PODS? Without this link, the mechanism explanation remains correlational.
Q5: Experiments are limited to math and code, where rewards are binary or test-case-based and response structure is relatively uniform. How would SSL behave on QA/search tasks or verifiable agent trajectories, where reward distributions, response lengths, and the structure of "useful negative signal" differ substantially? Is there evidence that the score-proxy and coalition utility generalize beyond math/code?
6 comments
Response to Reviewer hpLh (2/4)
W2b. Does D3S-Sample contradict the limitation of reward-statistic selection?
You correctly point out that full D3S includes cross-group allocation, entropy-based token selection, and a sampling schedule [1]. Table 2 evaluates that native system. D3S-Sample is the fixed-quota, within-prompt selection adaptation described in Appendix B.2.
Under this controlled condition, full-group normalization gives
=\frac{\operatorname{Var}_{i\in S}(r_i)}{(\sigma_N+\varepsilon)^2}.$$ The maximizing subsets therefore coincide with reward-variance maximizers up to ties in this specific setting. The following control isolates the additional information used by SSL by fixing its selected positive/negative reward counts: | Selector | Gradient cosine: subset normalization | Gradient cosine: full normalization | |---|---:|---:| | PODS / sign balance | 0.72 | 0.75 | | Random within SSL's reward composition | 0.66 | 0.72 | | SSL | **0.79** | **0.82** | At identical reward composition, SSL achieves higher fidelity by choosing responses whose directions better preserve the full-group signal. This is the contribution of joint directional valuation measured by the control. [1] Learning More with Less: A Dynamic Dual-Level Down-Sampling Framework for Efficient Policy Optimization. --- > #### **W2c / Q4. Do the same high-balance/high-conflict prompts show larger accuracy gains?** ${\color{green}\mathbf{Response.}}$ Yes, in the held-out cohort evaluated for this question. A common reference checkpoint defines the balance/conflict strata before method-specific training. The final SSL and PODS models are evaluated on those same 512 prompts with 32 responses per prompt and five training seeds. All strata, including the 16 degenerate prompts, are shown below. | Reference-defined region | Prompts | SSL−PODS fidelity difference (95% CI) | SSL−PODS final accuracy difference, pp (95% CI) | |---|---:|---:|---:| | $B>0.75,\\ C>0.65$ | 148 | **+0.1091 [0.085, 0.133]** | **+2.14 [1.25, 3.03]** | | $B>0.75,\\ C<0.40$ | 28 | +0.0290 [0.002, 0.056] | +0.60 [−0.40, 1.60] | | $B<0.40,\\ C>0.65$ | 6 | +0.0277 [−0.030, 0.086] | +0.40 [−2.10, 2.90] | | $B<0.40,\\ C<0.35$ | 43 | −0.0028 [−0.025, 0.019] | −0.10 [−0.70, 0.50] | | Other non-degenerate regions | 271 | +0.0509 [0.035, 0.067] | +0.55 [0.10, 1.00] | | Zero-reference/reward-degenerate | 16 | N/A | 0.00 | | All eligible regions | 496 | +0.0621 [0.048, 0.076] | +0.97 [0.60, 1.34] | | Prompt-level association | Estimate (95% CI) | |---|---:| | Partial Spearman between fidelity and accuracy differences, controlling benchmark and initial success | **0.26 [0.12, 0.39]** | This is a separate held-out cohort from Table 15's 2,483 eligible groups. A cloned-state intervention also tests actual optimizer behavior: | Method | AdamW displacement cosine to full GRPO | Frozen-objective improvement ratio after norm matching (95% CI) | |---|---:|---:| | PODS | 0.75 | 0.78 [0.73, 0.83] | | Best-pair greedy | 0.78 | 0.84 [0.79, 0.89] | | SSL | **0.83** | **0.90 [0.86, 0.94]** | The same high-balance/high-conflict prompts show the largest accuracy gain, as you asked. The partial correlation quantifies the association between fidelity and accuracy gains after controlling for benchmark and initial success. The cloned-state experiment provides complementary evidence that SSL preserves local improvement under actual AdamW updates.Response to Reviewer hpLh (3/4)
W2d / Q2. Does generation dominate, and how much does selecting 16 of 64 actually save?
You are right that all methods still generate 64 responses. Savings relative to full GRPO come from the policy update. The measured breakdown for the 3,072-token setting on eight A100s is:
| Method | Generation | Verification | Features + selection | Update + communication | Other | Total |
|---|---|---|---|---|---|---|
| Full GRPO, 64→64 | 265 | 14 | 0 | 260 | 6 | 545 |
| PODS, 64→16 | 265 | 14 | 2.0 | 64.0 | 6 | 351 |
| Best-pair greedy, 64→16 | 270 | 14 | 3.0 | 66.0 | 6 | 359 |
| SSL, 64→16 | 294 | 14 | 4.5 | 70.5 | 6 | 389 |
All entries are seconds per step. Generation accounts for 75.6% of SSL's total. Reducing the update workload still lowers end-to-end step time by 28.6% relative to full GRPO. SSL costs 10.8% more than PODS in this setting. That difference includes generation and update time as well as selection, because the independently trained SSL policy produces longer responses on average.
| Method | Mean response length | Generated tokens/update | Update tokens/update | Mean within 320 A100-GPU-h (%) |
|---|---|---|---|---|
| Full GRPO | 1,650 | 1,689,600 | 1,689,600 | 41.15 ± 0.58 |
| PODS | 1,650 | 1,689,600 | 407,040 | 41.80 ± 0.60 |
| Best-pair greedy | 1,680 | 1,720,320 | 422,400 | 42.38 ± 0.57 |
| SSL | 1,838 | 1,882,112 | 440,320 | 42.56 ± 0.55 |
Token totals cover 16 prompts per update. Greedy stops at 400 updates, consuming 319.1 GPU-hours within the 320-hour cap. In Table 1's short-response setting, SSL/PODS cost 5.2/4.8 GPU-hours, an 8.3% total increase. Selection accounts for 1.92% of SSL's time. The budget comparison includes generation, selection, and updates, and SSL has the highest mean accuracy under that cap.
Q3. Does the result transfer beyond Qwen, and what changes at 14B?
The original checkpoints overlap the evaluation settings of PODS, D3S/VIP, and CBS/ARRoL (Appendix B.4). Your family-transfer question is tested with Llama-3.1-8B-Instruct, using the extended mathematics protocol, 400 updates, and five seeds:
| Method | Gradient cosine | Math Mean ± SD (%) | Training GPU-hours |
|---|---|---|---|
| PODS | 0.71 ± 0.15 | 44.10 ± 0.58 | 310 |
| Best-pair greedy | 0.75 ± 0.14 | 44.32 ± 0.57 | 314 |
| SSL | 0.78 ± 0.13 | 44.61 ± 0.56 | 322 |
| Llama paired comparison | Mean difference (pp) | Reported paired 95% CI |
|---|---|---|
| SSL − PODS | +0.51 | approximately [0.08, 0.94] |
For your scale question, reward balance is highest at an intermediate success rate. With binary rewards, peaks at and is small near both zero and one. Conflict additionally depends on the signed directions. The frozen comparison on common prompts gives:
| Qwen2.5-Instruct model | Mean success rate | High-/high- group share | Proxy–gradient correlation | PODS / SSL cosine |
|---|---|---|---|---|
| 3B | 0.56 | 29% | 0.68 | 0.72 / 0.79 |
| 14B | 0.78 | 14% | 0.62 | 0.74 / 0.765 |
At 14B, the high-balance/high-conflict share falls from 29% to 14%, and SSL's fidelity advantage narrows from 0.07 to 0.025. The frozen diagnostic therefore supports your proposed dependence on the group's balance/conflict regime. The Llama experiment provides the complementary downstream training evidence across model families.
Response to Reviewer hpLh (1/4)
Thank you for taking the time to give such a detailed review and for your constructive suggestions. We address your questions below, focusing on the joint-selection baselines, the link between fidelity and accuracy, and transfer across models and tasks.
Response. A response's training value depends on the responses selected alongside it. Adding a response changes the coalition's reward center and signed direction, including the contributions of existing members. Shapley valuation captures this dependence by averaging each response's marginal contribution across coalition contexts. Greedy evaluates additions along one construction path. Your suggested comparison tests the practical value of that averaging.
All proxy methods below use the same frozen groups, d=128, and 64!→!16 quota. Best-pair initialization handles the zero-utility singleton tie explicitly.
The same ordering appears downstream, including after charging SSL's overhead to a common budget:
Greedy reaches its 400-update cap at 319.1 GPU-hours. SSL improves the mean by +0.50 pp at fixed updates and +0.18 pp within the common budget. The paired interval supports the fidelity advantage, and the downstream means show the same ordering after accounting for cost. This gives empirical support for averaging contributions across coalition contexts.
Response. The matched runs allow a direct test of your concern. A paired test uses the covariance between methods, which is absent from a calculation based only on their marginal SDs. The observed differences for Table 6 are:
All five differences are positive for each backbone. The paired-t intervals exclude zero under an approximately normal-difference assumption. Holm-adjusted p values are 0.0292 for all three comparisons. The exact two-sided sign-flip test gives p=0.0625, its smallest attainable value with five pairs. These inferential results apply to Table 6's aggregate score. Table 1 provides descriptive five-seed means for the short-response tasks.
For your GSM8K observation, Table 6 shows a +0.35 pp gain on Qwen2.5-Math-7B and small decreases on the other two backbones. The table below also includes all three AIME budget comparisons.
The seven-benchmark aggregate improves consistently across paired runs. The task-level results specify where that gain occurs, including the small GSM8K and AIME decreases shown above.