acceptodds
Under review as a conference paper at ICLR 2027

Train What Is Ready but Not Yet Mastered: Competence-Driven Curriculum Sampling for Reinforcement Learning with Verifiable Rewards

Abstract

Reinforcement learning with verifiable rewards (RLVR) is an effective paradigm for improving mathematical reasoning in large language models, yet the training prompt distribution is commonly determined by random sampling or manually specified curricula. Such schedules ignore that the useful training difficulty changes with the learner’s competence. We introduce a competence‑driven curriculum sampler that dynamically allocates each training batch across ordered difficulty regions according to two complementary signals: cumulative prerequisite readiness, which estimates whether the model has acquired the easier abilities required to approach a region, and mastery gap, which measures how much competence remains to be learned in that region. We maintain an exponentially smoothed success rate for each difficulty bucket, define readiness as the product of competence estimates over all prerequisite buckets, and assign sampling priority proportional to readiness times remaining mastery gap. This yields a closed‑loop curriculum whose progression is driven directly by model competence rather than training step. On DeepMath‑10K, the same scheduler achieves the highest eight‑benchmark mean among the compared sampling baselines at both Qwen3‑1.7B and Qwen3‑4B scales: 0.3992 versus 0.3944 for the strongest 1.7B baseline, and 0.5617 versus 0.5485 for the strongest 4B baseline. Difficulty‑range ablations further show that the best fixed range changes with model scale, while our adaptive scheduler remains best at both scales. These results support a simple principle for RLVR data selection: train what is ready to learn, but not yet mastered.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Large Language Models, RLVR, Curriculum Learning, Data Selection, Mathematical Reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_Vmt4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a lightweight curriculum sampler for RLVR. It maintains calibrated, EMA-smoothed success rates over ordered difficulty buckets and assigns sampling priorities using cumulative prerequisite readiness multiplied by the remaining mastery gap. Minimum quotas and largest-remainder allocation translate these priorities into training batches without changing the GRPO objective. Experiments on DeepMath-10K report the highest eight-benchmark mean among the evaluated baselines at both Qwen3-1.7B and Qwen3-4B scales.

Critical strengths
  1. The sampler is simple, interpretable, and lightweight, with the same scheduler hyperparameters used across both model scales.
  2. The evaluation includes relevant static and adaptive baselines, notably E2H schedules, ODF, and CDAS.
Critical weaknesses
  1. The ablations do not isolate cumulative readiness or online feedback from simpler difficulty reweighting.
  2. The prerequisite interpretation is not convincingly connected to useful GRPO learning signals.
  3. The improvements lack training-run randomness / uncertainty estimates.
  4. Some important baselines or related works are not well discussed or compared, leading to novelty concerns (see below).
Questions for authors

Based on the review above and some other issues, the reviewer has the following questions or comments.

  1. A1–A4 vary the available difficulty range but do not isolate the proposed readiness–mastery rule. Could the authors compare against mastery-gap-only, readiness-only, and frozen post-calibration quotas to establish whether cumulative readiness and online adaptation are necessary?
  2. Difficulty ordering does not itself establish that every easier bucket supplies prerequisites for every harder bucket, particularly across the heterogeneous mathematical topics in this dataset. With GRPO, A harder bucket can receive high priority once easier buckets are mastered, even if its own prompts still produce only zero rewards. Moreover, identical bucket-average success rates can conceal very different proportions of zero-advantage groups. Could the authors report per-bucket all-correct/all-incorrect group frequencies and test whether the proposed score tracks useful learning signals better than a bucket-local alternative?
  3. The improvements over ODF are 0.48 and 1.32 percentage points, without training-seed uncertainty estimates. Also, Random and A4 follow the same nominal sampling rule but obtain different results: 0.5335 versus 0.5263 at 4B. These two observations suggest the necessity for repeated training runs and randomness / uncertainty estimates.
  4. The difficulty-aware selection principle is closely related to DAPO’s Dynamic Sampling, which filters prompts with all-correct or all-incorrect rollout groups. Although DAPO is cited, its relationship to the proposed sampler is not discussed. How does it fit into the paper’s scheduler taxonomy, and what advantages does prerequisite-based allocation offer over such online filtering?
  5. SEC (Self-Evolving Curriculum) [1] is also relevant: it uses category-level EMA feedback for adaptive sampling. Discussing its different control signal would sharpen the novelty claim; an additional experiment would also be informative.
  6. The supplied code implements quota computation but omits the promised launch configurations, baseline implementations, and evaluation pipeline. Could these be provided, including the verifier and per-benchmark avg@16 aggregation code?

[1] Xiaoyin Chen, Jiarui Lu, Minsu Kim, Dinghuai Zhang, Jian Tang, Alexandre Piché, Nicolas Gontier, Yoshua Bengio, Ehsan Kamalloo: Self-Evolving Curriculum for LLM Reasoning. CoRR abs/2505.14970 (2025)

Reviewer_SFxLOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a curriculum learning method for RLVR that splits training prompts into difficulty buckets and gives each bucket a sampling priority equal to readiness times mastery gap. On DeepMath-10K, it achieves high accuracy among the baselines.

Critical strengths
  • The proposed method is simple.
  • The paper's writing is pretty clear.
Critical weaknesses
  • The design of the priority is not motivated at all. I found it hard to understand the motivation of prerequisite readiness, remaining learning need, and using the product of these two as the priority.
  • The method is designed specifically for one dataset, DeepMath-10K which is very very limited.
  • Definition of difficulty is often not monotonic and could change during training. For example, before training, the difficulty of some geometry problem is 8. But the model has seen a lot of geometry dataset during mid-training and it quickly picks up on them during RL. For the current model, those geometry problem would appear to be much easier than 8. Using pre-defined set of difficulty values appears suboptimal to me.
  • The theoretical properties add nothing new to understanding the curriculum. It does not help the reader to have a deeper understanding on the proposed curriculum.
  • The improvement on math benchmarks seem to be minimal, and might be within statistical noise of each benchmark.
Questions for authors

Please refer to the weaknesses

Reviewer_jSGEOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a bucket-level curriculum sampler for RLVR. The training set is split into four ordered difficulty buckets, and an EMA success rate is tracked for each. Each bucket’s sampling quota is set in proportion to the product of its “prerequisite readiness” (the product of the success rates of all easier buckets) and its “mastery gap” (one minus its own success rate). On DeepMath-10K with Qwen3-1.7B and Qwen3-4B, the method reports the best eight-benchmark mean among the compared samplers (0.3992 vs 0.3944 and 0.5617 vs 0.5485). A difficulty-range ablation shows that the best fixed range differs between the two scales.

Critical strengths

The method is simple, cheap and leaves the RL objective untouched, and the cross-scale difficulty-range ablation is a reasonable observation.

Critical weaknesses

The novelty is minimal. Online difficulty estimation for RLVR prompt selection was one of the most crowded topics of the past year. Bucket- or distribution-level adaptive curricula over difficulty levels already exist in SEC (Chen et al., 2025, Self-Evolving Curriculum for LLM Reasoning) and DUMP (Wang et al., 2025, Automated Distribution-Level Curriculum Learning for RL-based LLM Post-training). Adapting the target difficulty to the observed reward is the core of AdaRFT (Shi et al., 2025, Efficient Reinforcement Finetuning via Adaptive Curriculum Learning). Online prediction of prompt difficulty is done in MoPPS (Qu et al., 2025, Can Prompt Difficulty be Online Predicted for Accelerating RL Finetuning of Reasoning Models?). Prioritizing prompts by learnability is done in Foster & Foerster (2025, Learning to Reason at the Frontier of Learnability), on top of ODF, CDAS and DAPO’s dynamic sampling, which the paper itself cites. The underlying idea of choosing tasks by competence or learning progress goes back further still, to Platanios et al. (2019), Teacher-Student Curriculum Learning (Matiisen et al., 2017), ALP-GMM (Portelas et al., 2020) and Prioritized Level Replay (Jiang et al., 2021). The proposed score adds little to this line of work. For adjacent buckets it reduces to roughly pˉk−1(1−pˉk)p̄_{k-1}(1 − p̄_k), a one-bucket-shifted variant of the standard p(1−p)p(1 − p) learnability heuristic, wrapped in a product that is admittedly ad hoc. The “theoretical properties” are just signs of partial derivatives of a product and a textbook Lagrangian, and do not count as a contribution. None of the most directly comparable methods (SEC, DUMP, AdaRFT, MoPPS) are included as baselines.

Refs:

Bae, Sanghwan, et al. “Online Difficulty Filtering for Reasoning Oriented Reinforcement Learning.” Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computational Linguistics, 2026, pp. 700–19.

Chen, Xiaoyin, et al. “Self-Evolving Curriculum for LLM Reasoning.” arXiv, 2025.

Foster, Thomas, et al. “LILO: Learning to Reason at the Frontier of Learnability.” Advances in Neural Information Processing Systems, vol. 38, 2025.

Jiang, Minqi, et al. “Prioritized Level Replay.” Proceedings of the 38th International Conference on Machine Learning, edited by Marina Meila and Tong Zhang, PMLR, 2021, pp. 4940–50. Proceedings of Machine Learning Research 139.

Kong, Deyang, et al. “Rethinking the Sampling Criteria in Reinforcement Learning for LLM Reasoning: A Competence-Difficulty Alignment Perspective.” arXiv, 2025.

Matiisen, Tambet, et al. “Teacher–Student Curriculum Learning.” IEEE Transactions on Neural Networks and Learning Systems, vol. 31, no. 9, 2020, pp. 3732–40.

Platanios, Emmanouil Antonios, et al. “Competence-Based Curriculum Learning for Neural Machine Translation.” Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Association for Computational Linguistics, 2019, pp. 1162–72.

Portelas, Rémy, et al. “Teacher Algorithms for Curriculum Learning of Deep RL in Continuously Parameterized Environments.” Proceedings of the Conference on Robot Learning, edited by Leslie Pack Kaelbling et al., PMLR, 2020, pp. 835–53. Proceedings of Machine Learning Research 100.

Qu, Yun, et al. “Can Prompt Difficulty Be Online Predicted for Accelerating RL Finetuning of Reasoning Models?” arXiv, 2025.

Shi, Taiwei, et al. “Efficient Reinforcement Finetuning via Adaptive Curriculum Learning.” Transactions on Machine Learning Research, 2026.

Wang, Zhenting, et al. “DUMP: Automated Distribution-Level Curriculum Learning for RL-Based LLM Post-Training.” arXiv, 2025.

Yu, Qiying, et al. “DAPO: An Open-Source LLM Reinforcement Learning System at Scale.” Advances in Neural Information Processing Systems, vol. 38, 2025.

Questions for authors

N/A

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.