acceptodds
Under review as a conference paper at ICLR 2027

The Model is Its Own Curriculum: Internal Difficulty Assessment for Self-Adaptive RL in Reasoning

Abstract

Reinforcement learning can substantially improve language model reasoning, but sparse outcome rewards provide limited guidance for exploring complex reasoning trajectories. Curriculum learning offers a natural way to structure this exploration by adapting difficulty to the evolving policy. A key question, however, is at what granularity such learner-relative difficulty should be defined. While empirical pass rates provide a dynamic estimate of problem difficulty, the current policy's likelihood can further distinguish among correct reasoning trajectories for the same problem. We propose Self-Curriculum, which uses the negative length-normalized log-likelihood of correct trajectories as a learner-relative measure of trajectory difficulty. Self-Curriculum orders verified trajectories from easy to hard, periodically updates their difficulty as the policy evolves, and incorporates this signal into reward shaping. Across multiple mathematical reasoning benchmarks and model families, Self-Curriculum consistently outperforms standard GRPO. On Qwen3-8B, a curriculum based on dynamic pass rates already improves over GRPO, while Self-Curriculum yields further gains and also outperforms the evaluated external difficulty baselines at both the problem and trajectory levels. These results support policy likelihood as a fine-grained, learner-relative signal for adaptive curriculum learning under outcome supervision.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Model, Curriculum Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 2 reviews, above 85% of the others.
Reviewer_7mYuOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Self-Curriculum orders verified, self-generated solutions by the learner's length-normalized negative log-likelihood. It periodically reranks unused trajectories and gives less likely correct solutions higher rewards. Experiments cover mathematical reasoning, with additional model scaling and non-mathematical evaluations.

Critical strengths
  1. The controls are unusually helpful: student-generated pools are shared across difficulty signals, and filtering, ordering, reranking, and reward shaping are examined separately.
  2. Improvements survive independent training seeds and extend to 14B models. The reported overhead and expanded-rollout GRPO comparison make the practical case reasonably convincing.
Critical weaknesses
  1. The ranking remains sensitive to wording and notation despite length normalization. Figure 2 tracks likelihood on trajectories used in training, so it offers limited evidence that the score identifies a useful learning progression beyond fitting those particular strings.
  2. The custom NuminaMath split is described mainly through source counts. Duplicate or related contest problems are not explicitly grouped before splitting, and the AMC/AIME test subset has only 42 questions. The harder-math generalization evidence is therefore less secure than the main MATH results.
Questions for authors
  1. How stable are difficulty rankings under equivalent rewrites of the same correct solution?
  2. What deduplication or problem-family grouping was applied before constructing the NuminaMath train/test split?
Reviewer_3ABsOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Self-Curriculum, which orders self-generated correct responses using the model’s own trajectory likelihood. It periodically updates the ordering and gives higher rewards to lower-likelihood correct responses.

Critical strengths

The experiments cover multiple model families and include component ablations.

Critical weaknesses
  1. Limited novelty. Easy-to-hard training and adapting the curriculum to the learner’s progress are common approaches in reasoning RL [1, 2]. Rewarding low-probability but correct responses has also been explored [3]. The main extension here appears to be using trajectory likelihood for ordering and periodic reranking, rather than introducing a new curriculum learning idea.

  2. Dated and limited benchmarks. Evaluation relies on GSM8K, MATH, and a custom NuminaMath-CoT split. The “AMC/AIME” results cover only 42 questions from a mixed subset rather than a standard year-specific AIME benchmark.

    [1] Curriculum Reinforcement Learning from Easy to Hard Tasks Improves LLM Reasoning

    [2] Efficient Reinforcement Finetuning via Adaptive Curriculum Learning

    [3] Rewarding the Unlikely: Lifting GRPO Beyond Distribution Sharpening

Questions for authors
  1. Beyond combining existing curriculum and reward-shaping ideas, what methodological contribution distinguishes this approach from prior work?

  2. Do you have results on newer benchmarks, such as AIME 2026 and HMMT February 2026?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.