acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Surrogate Criteria for RLVR Rollout Filtering

Abstract

Reinforcement learning with verifiable rewards improves language-model reasoning, but it remains unclear whether common criteria for filtering generated responses preserve the learning signal needed for policy updates. We investigate when directional agreement preserves the gradient computed from all responses and when magnitude mismatch changes which subset should be preferred. We develop a controlled audit that separates gradient error into direction and scale, compares all subsets at the same retention budget, and evaluates selected subsets in the original trainable parameter space. Our theoretical analysis shows that perfect directional alignment can still yield large reconstruction error, motivating a selection criterion that jointly matches direction and magnitude. Across eight model–dataset pairs covering GSM8K, MATH, and TACO, retaining eight of sixteen responses with our criterion reduces mean squared relative gradient error by 66.8–75.7% compared with reward-variance selection and 42.9–53.7% compared with direction-only selection. These results identify scale mismatch as a limitation of common selection criteria and provide a framework for evaluating gradient preservation at a fixed policy.

Primary area:
foundation or frontier models, including LLMs
Keywords:
reinforcement learning with verifiable rewards, large language models, rollout filtering

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_ZkE1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies rollout filtering in GRPO-style RLVR from the perspective of gradient fidelity: given 16 sampled responses per prompt, which 8 should be kept so that the subset gradient best reproduces the full-group gradient at the current policy? The authors decompose the squared relative error into a scale term (r-1)^2 and a direction term 2r(1-c), show by construction (Theorem 1) that direction-only (cosine) selection can be arbitrarily worse than the error-optimal subset, and propose a joint-error selector that minimizes ||g_S - g_F||^2 in a 128-dimensional CountSketch of per-response score gradients. The evaluation is an offline, frozen-policy audit over all 12,870 size-8 subsets, evaluated exactly via a Gram matrix in rank-16 LoRA parameter space, on eight model-dataset pairs (Qwen2.5-Coder-3B, Qwen2.5-Math-7B, Llama-3.1-8B on GSM8K, MATH, TACO). Joint selection reduces mean squared relative error by 67-76% relative to PODS reward-variance selection and 43-54% relative to cosine selection, while cosine selection achieves the smallest angle; gains are larger when per-response contribution norms are heterogeneous and within-group cancellation is strong. Controls cover fixed vs recomputed advantages, sketch size, exact-geometry selection, and synthetic pairs.

Critical strengths
  1. The audit methodology is careful and the diagnostic finding is clean. The decomposition in Eq. (4) makes precise why cosine selection can match direction while leaving a large scale error (97.6% of cosine's error on Coder/GSM8K is norm mismatch), and the controls (exhaustive search, exact Gram evaluation, both advantage conventions, sketch size, synthetic pairs) rule out the obvious confounds. As a measurement study of what common filtering criteria preserve at a fixed policy, the paper is thorough and the writing of the method section is precise.
Critical weaknesses
  1. My main concern is whether lower gradient error helps training. Table 2 shows that joint selection mainly improves norm matching, while cosine does better after optimal rescaling. It is unclear whether this advantage survives minibatch aggregation and AdamW updates. I recognize the diagnostic scope, but an optimizer-aware test or a small training experiment is needed to establish practical relevance.

  2. The selector computes gradients for all 16 responses and searches 12,870 subsets before keeping eight. Reward-only methods avoid this gradient-extraction cost. This is reasonable for an offline audit, but the paper provides neither measured overhead nor a cheap approximation, leaving the implications for efficient filtering unclear.

Questions for authors
  1. Does joint selection still improve update fidelity after minibatch aggregation and AdamW updates, starting from the same optimizer state?

  2. Can you compare PODS, cosine, and joint selection in a small training run, reporting accuracy against compute?

  3. What is the measured selection overhead, and can it be amortized enough to yield net savings?

  4. Do the rankings and direction–scale trade-off persist at 16→4 and 16→12 retention?

Reviewer_4PNuOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper addresses the large resource consumption in RLVR training. The authors try to select a subset of rollout samples for training, aiming to recover the gradient obtained from the full set of rollouts.

In terms of experimental methodology, the authors conduct an offline audit on the frozen initial policy. They exhaustively search all 12,870 subsets of 8 out of 16 rollouts and evaluate the error exactly in the uncompressed LoRA parameter space.

The experiments cover 3 models and 3 datasets (GSM8K, MATH, and TACO). The squared relative error of the joint error criterion is lower than that of the PODS variance criterion

Critical strengths

The paper decomposes the gradient error into magnitude and direction components for its audit, and analyzes how to better approximate the gradient of the full set of rollout samples.

Critical weaknesses

The contribution of this paper is not evident from either the theoretical or the experimental perspective.

  1. The method is not usable in practice. Both the joint error criterion and the cosine criterion require first computing the gradients of all n responses and the full-group gradient before selecting k of them. Once full-group gradient has been computed, one can simply use it directly for the update with zero error. Selecting a subset to approximate it is therefore meaningless.
  2. The motivation is severely limited. In long-trajectory RL training, rollout generation usually accounts for the majority of training time, so the overall speedup this kind of filtering can provide is limited. The paper also does not provide any analysis of gains in time or compute.
  3. No training experiments. All results are single-step gradient errors on the frozen initial policy. There are no RL training curves or efficiency comparisons. Even if the paper’s contribution were restricted to short-context RL training, this would still be inadequate.
Questions for authors

None

Reviewer_iBJAOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

In this paper, the authors investigate whether commonly used surrogate criteria for RLVR rollout filtering preserve the full-group policy gradient. The authors argue that directional alignment alone is insufficient for gradient fidelity, as a selected subset may closely match the direction of the full-group gradient while exhibiting substantial magnitude mismatch. To address this issue, the paper introduces a joint-error selection criterion that explicitly considers both gradient direction and magnitude.

Critical strengths
  1. The paper is well-written and easy to follow;

  2. The authors provided an RLVR-realizable counterexample in which cosine-based selection produces arbitrarily worse reconstruction error relative to the optimum.

  3. The experiments cover several LLMs and reasoning benchmarks.

Critical weaknesses
  1. Table 2 and Table 9 report gradient reconstruction errors for Qwen2.5-Coder-3B-Instruct on GSM8K, but their numerical results differ substantially.

  2. The paper motivates rollout filtering as a means of reducing the computational cost of RLVR training. However, the proposed joint-error selection procedure requires computing score gradients for all generated responses before selecting a subset. The authors overclaim their contributions.

  3. The theoretical results are primarily established under exact gradient geometry and specific assumptions. In contrast, the main experiments employ CountSketch-based gradient approximations for subset selection. The current theoretical analysis does not directly establish reconstruction guarantees for this approximate selection procedure. Moreover, the analysis focuses on fixed-policy gradients, while practical RLVR training involves continuously changing policies and multiple optimization steps.

  4. The proposed joint-error criterion is closely related to the gradient reconstruction objectives already studied in the coreset and data subset selection papers, like Grad-Match and CRAIG.

  5. The empirical evaluation focuses on gradient reconstruction error at fixed policy states, without demonstrating that improved reconstruction fidelity leads to better downstream performance.

Questions for authors

Please see Weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.