- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies rollout filtering in GRPO-style RLVR from the perspective of gradient fidelity: given 16 sampled responses per prompt, which 8 should be kept so that the subset gradient best reproduces the full-group gradient at the current policy? The authors decompose the squared relative error into a scale term (r-1)^2 and a direction term 2r(1-c), show by construction (Theorem 1) that direction-only (cosine) selection can be arbitrarily worse than the error-optimal subset, and propose a joint-error selector that minimizes ||g_S - g_F||^2 in a 128-dimensional CountSketch of per-response score gradients. The evaluation is an offline, frozen-policy audit over all 12,870 size-8 subsets, evaluated exactly via a Gram matrix in rank-16 LoRA parameter space, on eight model-dataset pairs (Qwen2.5-Coder-3B, Qwen2.5-Math-7B, Llama-3.1-8B on GSM8K, MATH, TACO). Joint selection reduces mean squared relative error by 67-76% relative to PODS reward-variance selection and 43-54% relative to cosine selection, while cosine selection achieves the smallest angle; gains are larger when per-response contribution norms are heterogeneous and within-group cancellation is strong. Controls cover fixed vs recomputed advantages, sketch size, exact-geometry selection, and synthetic pairs.
Critical strengths
- The audit methodology is careful and the diagnostic finding is clean. The decomposition in Eq. (4) makes precise why cosine selection can match direction while leaving a large scale error (97.6% of cosine's error on Coder/GSM8K is norm mismatch), and the controls (exhaustive search, exact Gram evaluation, both advantage conventions, sketch size, synthetic pairs) rule out the obvious confounds. As a measurement study of what common filtering criteria preserve at a fixed policy, the paper is thorough and the writing of the method section is precise.
Critical weaknesses
-
My main concern is whether lower gradient error helps training. Table 2 shows that joint selection mainly improves norm matching, while cosine does better after optimal rescaling. It is unclear whether this advantage survives minibatch aggregation and AdamW updates. I recognize the diagnostic scope, but an optimizer-aware test or a small training experiment is needed to establish practical relevance.
-
The selector computes gradients for all 16 responses and searches 12,870 subsets before keeping eight. Reward-only methods avoid this gradient-extraction cost. This is reasonable for an offline audit, but the paper provides neither measured overhead nor a cheap approximation, leaving the implications for efficient filtering unclear.
Questions for authors
-
Does joint selection still improve update fidelity after minibatch aggregation and AdamW updates, starting from the same optimizer state?
-
Can you compare PODS, cosine, and joint selection in a small training run, reporting accuracy against compute?
-
What is the measured selection overhead, and can it be amortized enough to yield net savings?
-
Do the rankings and direction–scale trade-off persist at 16→4 and 16→12 retention?