acceptodds
Under review as a conference paper at ICLR 2027

Same Mastery, Opposite Directions: Signed Learning Progress for Prompt Revisiting in RLVR

Abstract

While reinforcement learning with verifiable rewards has improved reasoning, the dominant cost is grouped on-policy generation, and most methods still spend a fixed group budget according to a prompt's current state. We study this allocation through learning progress, the recent change in pass rate relative to a slower mastery state. A comparison of uniformly sampled GRPO revisits uncovers two facts. First, current difficulty, competence match, and the magnitude of historical advantage assign the same priority to two prompts with the same pass rate, whether the policy is improving or regressing, while an all-correct or all-incorrect group has zero relative advantage. Second, with mastery and the size of the stored change held fixed, the next visit raises the pass rate by 22.4 points when that change is positive and by 1.1 points when it is negative, so another group is better spent on the improving prompt. Based on these observations, we propose LP-GRPO, which reads a slow mastery state and a fast signed progress state from before the current group. The state is used twice: LP-Sampling spends the next groups where a nonzero advantage is still available and progress is positive, and LP-Reweighting multiplies the resulting nonzero advantages by a positive weight from the same state. Neither step adds a selector or a value model. On eight multimodal benchmarks, with the same number of retained groups per update, LP-GRPO raises GRPO from 66.50 to 69.75 and exceeds the strongest matched baseline by 1.62 points. The revisit rule alone reaches 69.18. The full method matches GRPO's final average in half the updates, and CDAS and PSPO in two thirds.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Reinforcement Learning for LLMs;Sample Efficiency;RLVR

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_6Tq5OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors propose LP-GRPO, a method that decides which prompts get the next GRPO rollout groups. It keeps two scalars per prompt: a slow EMA of the pass rate (“mastery”) and a fast EMA of the signed gap between the new pass rate and that mastery (“learning progress”, LP). The motivation is that difficulty-based schedulers give two prompts with the same pass rate the same priority, even when one is improving and the other is regressing. On 14,795 revisits from uniform GRPO, the authors report that the next visit raises the pass rate by 22.4 points when stored LP is positive and by 1.1 points when it is negative (Section 1).

LP-GRPO has two parts. LP-Sampling samples prompts in proportion to 4 hm(1−hm)\sqrt{4\,\mathrm{hm}(1-\mathrm{hm})}, multiplied by (1+λ[lp]+)(1+\lambda[\mathrm{lp}]_+) so that only positive progress gets a boost (Eq. 8). LP-Reweighting multiplies each prompt’s GRPO advantages by a positive weight built from the offline pass rate and the stored LP (Eqs. 11 to 14). Both are meant to read only state stored before the current group is generated.

With 128 retained groups per step and 360 steps, LP-GRPO reaches 69.75 on eight multimodal benchmarks with Qwen3-VL-8B-Instruct. GRPO reaches 66.50 and CDAS 68.13 (Table 1). LP-Sampling alone reaches 69.18 (Table 2). Sampling with positive LP, no LP, and negative LP gives 69.18, 68.29, and 66.89 (Figure 5). The paper also reports 58.20 on Qwen2.5-VL-7B (Table 1) and 44.86 on a text-only Qwen2.5-7B protocol (Table 5).

Critical strengths
  1. The method is cheap and easy to adopt. It stores a few scalars per prompt and adds no rollouts, no selector model, and no value model (Section 3.3). Sampling reads only state written through step t−1t-1, so a group’s own reward cannot raise its own chance of being picked. This matters to my assessment because even a reliable gain of one or two points at no extra cost would be useful to anyone running GRPO.
  2. The direction ablation is well controlled. Figure 5 compares positive LP, no LP (λ=0\lambda=0), and negative LP with the same eligibility rule, batch size, and optimizer. The order is the same on all eight benchmarks and at every checkpoint, ending at 69.18, 68.29, and 66.89. This is the most direct evidence in the paper that the sign of recent progress carries information beyond the current pass rate. It is the main reason I am not recommending a clear rejection.
Critical weaknesses
  1. The evidence does not separate signed progress from noise or from simpler explanations. Each method is trained once (Appendix A.2: “Each trained checkpoint is from one run”). Each benchmark is scored with a single greedy decode at temperature 0 (Section 4.1). So n = 1 for training runs and n = 1 sample per question. The gaps behind the main claim are small: 1.62 over CDAS (Table 1), 0.89 between positive LP and no LP (Figure 5), and 0.57 from adding reweighting to LP-Sampling (Table 2). Several test sets are small, including LogicVista and the testmini/MINI splits of MathVista and MathVerse. Headroom is also uneven. MathVista starts at 77.20 for the base model, and the best matched run reaches only 80.40 (Table 1). LP-GRPO is not the best matched method on MathVista or MMK12.

There is also a simpler explanation for much of the gain. The pool keeps 10,000 prompts that the base model solved 0 of 8 times, about a third of the 30,445 prompts (Appendix A.1). GRPO spends 29% of its groups on nearly unsolved prompts (Figure 1). Under LP-Sampling these prompts score about ρ0=0.15\rho_0 = 0.15, while mid-difficulty prompts score near 1. Mastery-only sampling already moves the average from 66.50 to 68.29 (Figure 5). The paper does not compare against absolute LP, the usual choice in the learning-progress literature (Graves et al., 2017; Portelas et al., 2020; Kanitscheider et al., 2021). It also does not report the constant-scale, difficulty-only, and progress-only reweighting controls promised in Section 3.3. I do not expect a static filter to match LP-GRPO. My point is that without these controls I cannot tell how much of the 3.25-point gain comes from the sign of progress, and the sign of progress is the claim in the title.

What would fix this: 3 training seeds for GRPO, CDAS, LP-Sampling, and LP-GRPO with average pass@1 over at least 8 samples, plus GRPO on the 1/8 to 7/8 subset, absolute-LP sampling, and the reweighting split.

  1. The motivating statistic and the mechanism analysis may not show what the paper says they show, and the paper describes the method in two different ways. The 22.4 vs 1.1 result (Section 1) does not say what the gain is measured against. The 142 strata are not described anywhere. If the gain is the next pass rate minus mastery, I would expect a gap like this even with no learning: mastery is a lagged, noisy estimate, and positive LP means the recent pass rates were above it. Proposition 3 already gives a stationary noise model that could serve as the null. Table 3 has a similar issue. It follows only prompts that reach a fourth visit, and the sampler favors prompts whose pass rate went up, so the 21.9-point gain on hard prompts is partly survivorship. It also has no GRPO column.

Separately, Eq. (10) and Section 3.3 say that first-visit prompts get LP = 0 for reweighting, so that the method uses only state stored before generation. Appendix B.4 and step 5 of Algorithm 1 use pt−p0p_t - p_0 on a first visit instead, which lets the current group set its own weight. At step 40 only 5 of 128 groups use stored LP (Table 4), so this rule sets the weight of most groups early in training.

What would fix this: the exact definition of the 22.4/1.1 statistic, the same numbers under a stationary null, a GRPO column for Table 3, and a statement of which reweighting rule produced the reported runs.

Questions for authors
  • How were the method’s 13 hyperparameters chosen? Table 10 covers only MathVision, and no validation set is mentioned. Table 6 lists a 3-epoch schedule, but all methods are compared at step 360, about 1.5 epochs, and the curves in Figure 3 are still rising. * What is the learning-rate schedule? Does “matches GRPO’s final average in half the updates” refer only to step 360?
  • PSPO is listed as a baseline (Section 4.1) and appears in Figure 3, but not in Table 1. Why?
  • In Table 5, Random and Curriculum Sampling have identical scores on all six benchmarks, and the PSPO row is a published Qwen2.5-* Math-7B result. Were these baselines rerun? The AIME scores (e.g., 12.08) are not multiples of 1/30. Is the text-only evaluation averaged over several samples, unlike the multimodal one?
  • Do the Qwen2.5-VL-7B baselines taken from Huang et al. (2026) use the same data, rollout count, and step budget as your LP-GRPO run?
  • Minor: in Table 1, VPPO’s 80.40 on MathVista should be bold. Line 221 says “the ICLR formulation”. Figure 2 drops the [⋅]+[\cdot]_+ from Eq. (8). Figure 1 is hard to read. Related work should discuss Foster and Foerster (2025) on p(1−p)p(1-p) learnability sampling, as well as GRESO, SEC, and Online Difficulty Filtering.
Reviewer_KKTbOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies how GRPO-style RLVR should spend a fixed number of retained prompt groups per update. It argues that difficulty-based samplers give the same priority to two prompts with the same mastery even when one is improving and the other is regressing. As evidence, it reports that under uniform GRPO, prompts with positive stored progress gain far more on their next visit than prompts at the same mastery with negative progress. LP-GRPO keeps two states per prompt, a slow mastery average and a fast average of the residual from that mastery. Both are read before the current group is drawn. LP-Sampling favors prompts with intermediate mastery and boosts those with positive progress. LP-Reweighting scales the GRPO advantages by a positive weight built from the offline pass rate and the progress state. On Qwen3-VL-8B-Instruct and eight multimodal benchmarks, LP-GRPO raises the average of GRPO from 66.50 to 69.75. Most of the gain comes from the sampler. Further results cover Qwen2.5-VL-7B and a text-only protocol.

Critical strengths
  1. A relevant problem with a cheap design.

    All-correct and all-incorrect groups waste generation in GRPO, so where to spend a fixed group budget is a practical question. LP-GRPO keeps a few scalars per prompt and adds no selector, value model or extra online rollout (Section 3.3). It only changes which prompts are drawn and how their advantages are scaled, so it would be easy to add to existing GRPO pipelines.

  2. A careful 8B comparison with checkpoint curves and component ablations.

    All trained 8B methods share the base model, prompt pool, group size, learning rate and update budget (Section 4.1). Figure 3 reports every benchmark at six checkpoints rather than only at the end. Table 2 separates sampling from reweighting, and Figure 5 contrasts positive, zero and negative progress boosts under the same eligibility rule.

  3. The prompt state is analyzed and logged in unusual detail.

    Propositions 1 to 3 characterize the bounds of the two-timescale state, its response to a level shift and its stationary noise floor. Table 4 and Figure 12 track coverage, the share of fresh prompts, selection pressure and the use of stored progress throughout training. This makes the behavior of the sampler easy to inspect and to check against its description.

Critical weaknesses
  1. The central claim confuses the level of the pass rate with its direction.

    The title claim rests on the revisit analysis in the Introduction, which compares the two signs of stored progress at fixed mastery. Mastery is a slow average, so at equal mastery positive progress also means that recent pass rates were above it and negative progress means they were below it. The statement that "the current pass rate is the same" therefore does not hold. Each pass rate also comes from only eight rollouts, so part of any stored change is luck that disappears on the next visit. Depending on the reference point of the gain, which the paper never defines, this regression alone can produce a large gap in either direction without any learning. Even a real gap would not justify the allocation rule, because the next-visit pass rate is measured before that visit's group is trained on and does not show what an extra group contributes. RQ2 inherits the confound, since its variants differ only in the boost on top of the same slow mastery and none uses a fresher estimate such as the latest pass rate.

  2. The progress signal is mostly sampling noise where the method uses it.

    Each pass rate comes from eight rollouts, and even at step 360 seen prompts average fewer than two visits (Table 4). By Proposition 3, a prompt at an intermediate pass rate that is not learning at all has a progress standard deviation larger than the shift that one extra success in a group produces (Appendix B.3). The remark after Proposition 3 itself states that an isolated positive or negative value is not proof of learning or forgetting. Yet the sampler boosts a prompt after a single positive change. Appendix C.1 shrinks the gain for negative progress because one downward change is a noisy sign of forgetting, and it contrasts this with "repeated evidence" of improvement. The method never asks for such evidence, and one upward change is just as noisy. Apart from the confounded analysis in Weakness 1, nothing in the paper shows that the stored sign tracks a lasting trend.

  3. More than half of the gain comes from offline difficulty targeting rather than from progress.

    The Introduction calls the shift of groups toward intermediate pass rates "the core motivation" of the allocation strategy (Figure 1), a shift that difficulty-based samplers such as CDAS already make. The variant without progress matches CDAS and PSPO and yields more than half of the gain over GRPO (Figure 5). Its difficulty estimates rest largely on offline pass rates. Fresh prompts are scored by them alone and still fill two thirds of each batch at step 120 (Table 4), and the slow mastery update keeps the offline value dominant after one or two visits. Only LP-GRPO receives these per-prompt estimates (Appendix A.1), which come from a profiling pass comparable in cost to a full training run. The claim of matching GRPO in half the updates leaves this cost out, and no reported run initializes mastery from the first online visit as Section 3.1 allows. How much of the gain comes from offline information rather than from learning progress is therefore unknown.

  4. The reweighting used in the experiments is not the one described.

    Eq. 10 gives fresh prompts a neutral progress factor, and Section 3.3 states that a group's own outcome cannot raise its progress factor. Algorithm 1 and Appendix B.4 instead set first-visit progress from the current group's pass rate minus the offline one, which is exactly what Section 3.3 rules out. The diagnostics follow the appendix rule. Figure 12(c) shows an average weight above one during the first 30 updates, when every group is fresh, while Eq. 10 keeps every fresh weight below one. Fresh prompts fill most early batches, so a large part of the LP-Reweighting effect comes from this same-group rule rather than from stored progress. Table 3 also reports an average weight for easy prompts that is below the minimum the stated configuration allows.

Questions for authors
  1. What reference point defines the next-visit gain in the revisit analysis? Does the gap between the two signs remain when the latest pass rate is also held fixed within each stratum?

  2. How does the sampler perform without the boost when its difficulty term uses the latest pass rate or a faster mastery average? How does absolute progress compare with rectified progress in the sampler?

  3. How does LP-GRPO perform when mastery is initialized from the first online visit instead of the offline pass rate?

  4. Did the reported runs use Eq. 10 or the first-visit fallback of Algorithm 1? How is the average weight of easy prompts in Table 3 obtained under the stated configuration?

Reviewer_ffnpOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes LP-GRPO, a method that improves rollout allocation in RLVR by tracking both historical mastery and recent learning progress for each prompt. It consists of two components: LP-Sampling, which prioritizes prompts with intermediate mastery and positive learning progress, and LP-Reweighting, which adjusts their advantages using the stored learning state. The method does not require an additional model or extra online rollouts. Experiments on eight multimodal reasoning benchmarks show that LP-GRPO outperforms GRPO and several sampling baselines under a matched rollout-group budget. The authors also provide ablations and analyses of prompt revisiting, learning direction, and training efficiency.

Critical strengths
  1. A useful perspective on prompt selection. The paper makes an interesting distinction between a prompt's current mastery and its recent learning direction. Two prompts with similar success rates may have different training value, and using this information for rollout allocation is a reasonable idea.

  2. Simple and practical method. LP-GRPO maintains only a few statistics for each prompt and does not introduce an additional model. The method can be incorporated into the existing GRPO training pipeline with limited modifications.

  3. Good empirical performance. On Qwen3-VL-8B, LP-GRPO improves the average accuracy from 66.50 to 69.75 across eight benchmarks. The comparisons with CDAS, DAPO, and other methods are useful, and the improvements on both Qwen3-VL and Qwen2.5-VL provide some evidence of generalization.

  4. The sampling ablations are particularly useful. Table 2 shows that LP-Sampling accounts for most of the improvement, while Figure 5 compares positive-progress, mastery-only, and negative-progress sampling. These experiments help explain why the proposed sampling rule works better than simply prioritizing prompts based on difficulty.

Critical weaknesses
  1. The interpretation of positive learning progress needs more support. The method favors prompts whose recent pass rate exceeds their historical mastery. However, pass rates are estimated from only eight responses, so the progress signal can be noisy. A positive change may reflect sampling fluctuations rather than genuine improvement. The paper provides a revisit analysis, but it would be helpful to understand how reliably the estimated progress predicts future learning.

  2. The computational efficiency comparison is not entirely complete. The methods are compared using the same number of retained rollout groups, which is reasonable. However, LP-GRPO also uses historical prompt information, and the cost of obtaining initial pass rates should be accounted for. Furthermore, equal numbers of groups do not necessarily imply equal generation costs, especially if the sampled prompts produce responses of different lengths. A comparison using total generated tokens or GPU hours would be more convincing.

  3. The method introduces several hyperparameters. LP-GRPO requires two smoothing rates, a progress coefficient, difficulty and progress weighting parameters, and revisit constraints. Their interactions may affect the sampling distribution considerably. While the appendix includes some sensitivity experiments, it is not clear whether the same configuration works reliably across different training settings.

  4. The additional benefit of LP-Reweighting is relatively small. In Table 2, LP-Sampling alone achieves 69.18, while the full method reaches 69.75. On some benchmarks, including DynaMath and MathVerse, the sampling-only variant performs slightly better. This suggests that prompt selection is the main contribution, while the advantage reweighting component is less convincing. The authors could better explain when reweighting is beneficial.

  5. The experiments would benefit from stronger statistical evidence. The main matched-training results appear to rely on single training runs, without reporting variability across random seeds. Since the improvement over the strongest baseline is 1.62 points, it would be useful to establish how stable this difference is. The text-only results are also less impressive, with LP-GRPO improving over CDAS by only 0.35 points.

Questions for authors
  1. How reliable is the estimated learning-progress signal when each prompt is evaluated using only eight responses? Have the authors considered whether the observed positive progress could partly come from sampling noise?

  2. Does LP-GRPO maintain its efficiency advantage when comparing total generated tokens or actual training time, rather than only retained rollout groups? Does the reported budget include the cost of initializing prompt mastery?

  3. How sensitive is the method to the smoothing rates and revisit constraints? In particular, would the same hyperparameters work on datasets with different sizes or difficulty distributions?

  4. Table 2 shows that LP-Sampling alone achieves most of the improvement. Could the authors explain why LP-Reweighting provides only a relatively small additional gain and sometimes reduces performance on individual benchmarks?

  5. Can the authors report results across multiple random seeds, particularly for the comparison with CDAS? This would help assess whether the relatively small performance differences are consistent.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.