acceptodds
Under review as a conference paper at ICLR 2027

MoRe: Sensing Prompt Utility Through Gate Modulation for Sample-Efficient RLVR

Abstract

Reinforcement learning with verifiable rewards (RLVR) must allocate a limited rollout budget across prompts. We observe systematic differences in full-parameter gradient energy across prompts, even after accounting for observed success counts and prompt length. To characterize one model-internal source of this variation, we introduce modulation reactivity (MoRe), a closed-form measure of final-layer prompt-cache sensitivity to gate modulation that preserves bilinear interactions. MoRe uses only current weights and prefill activations, requiring no additional rollouts or backward passes. The same Jacobian connects cache responses to gradient feedback, giving MoRe a normalized expected gradient-energy interpretation under isotropic reference feedback. Among prompts matched on observed success rate and length, the high-MoRe group exhibits up to twice the cache-path feedback energy of the low-MoRe group. Fixing within-prompt attention proportions removes more than half of this matched feedback gap without changing MoRe. These findings motivate a two-stage sampler that uses reward beliefs to retain prompts likely to yield mixed rewards, then allocates a fixed number of rollout slots using current MoRe scores. On mathematical reasoning benchmarks, the sampler improves average accuracy by 1.92 percentage points over vanilla GRPO.

Primary area:
reinforcement learning
Keywords:
RLVR, LLM, Data Selection, Mechanistic Interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_tjkPOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MoRe, a prompt utility metric that measures KV-cache sensitivity to gate modulation without additional rollouts or backward passes. The authors establish a theoretical connection between MoRe and gradient feedback energy, supported by empirical and mechanistic analyses. Building on this metric, they develop a two-stage prompt sampling strategy that combines reward-based filtering with MoRe-guided selection. Experiments on mathematical reasoning tasks demonstrate improved sample efficiency and accuracy over existing baselines.

Critical strengths
  1. The paper addresses sample-efficient prompt selection in RLVR by considering model-internal signals beyond reward history.

  2. MoRe provides a mathematically grounded measure of prompt sensitivity through gate modulation, without requiring additional rollouts or backward passes.

  3. The paper investigates the relationship between MoRe and gradient feedback through matched comparisons and attention interventions.

Critical weaknesses
  1. MoRe measures Jacobian sensitivity and is empirically associated with gradient feedback energy. However, larger gradient energy does not necessarily imply more useful policy updates, as it may simply reflect higher gradient variance.

  2. The theoretical connection between MoRe and gradient energy relies on isotropic reference feedback, whereas actual GRPO feedback can be highly anisotropic.

  3. The experiments are limited to relatively small models (up to 4B parameters) and primarily focus on mathematical reasoning tasks. Moreover, the analysis connecting MoRe to gradient feedback is conducted on a single 0.5B model. Evaluations on larger models and more diverse RLVR tasks, such as code generation and agentic tasks, would help establish the generalizability of the proposed metric.

Questions for authors

MoRe only measures sensitivity along gate-modulation directions through the final-layer prompt KV-cache. While this makes the metric computationally tractable, it is unclear whether this restricted subspace is representative of the full-parameter gradients used in RLVR. Comparisons with other layers and parameter subspaces would strengthen the motivation for this design.

Reviewer_p8MFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies prompt selection for reinforcement learning with verifiable rewards (RLVR) under a fixed rollout budget. It proposes modulation reactivity (MoRe), which quantifies the sensitivity of the prefill KV-cache without rollouts or backward passes. On a 0.5B model with GSM8K, high-MoRe prompts show about twice the gradient feedback energy of low-MoRe prompts in the later training stage. Finally, the paper combines MoRe with MoPPS reward beliefs in a two-stage sampler and presents an empirical analysis of the proposed module on six math benchmarks with different model sizes.

Critical strengths
  • The main strength is a new and cheap signal for prompt selection in RLVR. Given the difficulty in prompt selection in RLVR, the new algorithm has the opportunity to be highly additive to the field.
Critical weaknesses
  • My main concern is the scale of experiments. The mechanism analysis is done only on a 0.5B model and GSM8K and the tests on math benchmarks are done with a maximum of 4B model. It is unclear whether MoRe still reflects gradient feedback on larger models or whether the gains hold at 7B or above.
Questions for authors
  1. What are the results when tested on larger models, say 7B or above? We can notice that gains are much more consisten for 4B model. It is not clear to me why this happens and whether results will be similar when tested on larger models.
  2. Since the result algorithm combines MoPPS, are there any experiments on the performance of the algorithm using only information from MoRe so that we can see which module contributes more to the performance?
Reviewer_SoRtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies prompt selection for RLVR. It introduces modulation reactivity (MoRe), a closed-form measure of how the final-layer prompt KV cache responds to normalized gate perturbations. It is computable before generation and without additional rollouts or backward passes. The authors connect MoRe to cache-path gradient energy, investigate the role of attention, and combine the score with reward-history filtering to allocate rollout budgets. The analytical construction is distinctive, the presentation is generally clear, and the mechanistic experiments are carefully controlled.

Critical strengths
  1. Reciprocal gate/up scaling preserves each channel's bilinear interaction while isolating changes in gate gain. Normalizing the parameter directions makes comparisons across prompts well defined. The resulting closed-form cache Jacobian norm provides an interpretable alternative to purely empirical selection heuristics.

  2. The diagnostic study separates analysis-development prompts from evaluation prompts and uses independent answer samples for estimating success rates and measuring feedback. Matching on success counts and prompt-length strata strengthens the finding that high-MoRe prompts have approximately twice the cache-path feedback energy at step 100.

  3. Results show improvements across the three evaluated models. Table 1 reports higher average mathematical accuracy than all listed baselines for each model, with gains over uniform sampling.

Critical weaknesses
  1. Higher feedback energy does not yet establish higher learning utility. Actual feedback can be anisotropic, and its second moment includes sampling variance. Appendix C.1 appropriately examines this distinction, but the high/low ratio for squared mean feedback has a wide confidence interval. Thus, the evidence does not resolve whether MoRe identifies stronger useful gradients or greater variability. My suggestion is to directly measure subsequent reward improvement or full-gradient alignment, especially beyond the single diagnostic model and dataset.

  2. Relative to MoPPS, the method changes pool construction, final allocation, and the finite-group normalization correction. Reversing the score ordering does not establish the benefit over uniform selection within the identical retained pool. Ablations should separately remove MoRe, the correction, and online score recomputation.

  3. All main training results use one random seed, and checkpoints are selected by maximizing the same six-benchmark average subsequently reported. With gains of only 0.50–1.10% over the strongest listed baseline for each model, training variation and checkpoint-selection noise could materially affect the conclusions. Multiple training seeds and checkpoint selection on an independent validation set are needed.

  4. Costs are reported at each method's accuracy-maximizing checkpoint, which does not establish performance at a common resource budget. For Qwen3-0.6B, the proposed method uses 34 GPU-hours versus 29 for Uniform and 27 for MoPPS. For Qwen3-4B, it uses more rollout tokens than Uniform, although fewer GPU-hours. Please consider to draw accuracy-versus-token and accuracy-versus-GPU-hour curves, together with a breakdown of candidate-prefill and score-computation overhead.

Questions for authors

NA

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.