acceptodds
Under review as a conference paper at ICLR 2027

CLPO: Curriculum Learning Meets Policy Optimization for LLM Reasoning

Abstract

Online reinforcement learning with verifiable rewards (RLVR) has become an effective paradigm for improving the reasoning abilities of large language models, but most methods still optimize reasoning trajectories over the static problem set, wasting rollout budget on solved or overly difficult problems. We propose CLPO (Curriculum Learning meets Policy Optimization), a self-evolving curriculum framework that uses on-policy rollout accuracy to identify solved, medium-difficulty, and hard problems, then restructures selected tasks according to the model's current capability. Hard problems are simplified to become learnable, while medium-difficulty problems are diversified to provide useful training variation. This allows the learning curriculum to co-evolve with the policy rather than remaining fixed as the model's capability boundary shifts. Rather than treating these rewrites as static data augmentation, CLPO optimizes restructuring trajectories with credit assigned by the downstream accuracy gain of the rewritten problem, requiring no additional human annotations beyond the original verifiable answers. Experiments across mathematical reasoning and out-of-domain general reasoning benchmarks show that CLPO substantially outperforms GRPO and DAPO on Qwen3-8B by 10.21 and 7.75 average points, respectively. Ablation studies on math and code domains further show that both the restructuring mode and the rewriting loss contribute to the final gains, demonstrating that CLPO provides a scalable and robust pathway for eliciting stronger reasoning capabilities through a self-evolving curriculum.

Primary area:
reinforcement learning
Keywords:
LLM, reasoning model, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_xpkXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to make RLVR training data better match the model's current reasoning ability. The authors point out that most RLVR methods train on a fixed set of problems. However, as the model improves, some problems may become too easy or remain too hard, and thus provide little useful learning signal. To address this, the authors propose CLPO, which dynamically adjusts training problems based on the model's rollout accuracy. It first divides problems into hard, medium, and easy groups. Hard problems are simplified, while medium problems are diversified. The authors also introduce a restructuring-aware objective, which uses the accuracy difference before and after rewriting (ΔAcc) to reward useful rewrites. The model is then jointly trained on original problems, rewritten problems, and rewriting trajectories. Experiments on Qwen3-8B show that CLPO achieves an average accuracy of 72.25% across eight benchmarks, outperforming GRPO, DAPO, and Critique-GRPO (CoT) by 10.21, 7.75, and 5.48 points, respectively. The authors also provide ablations, fixed-time training comparisons, and experiments on different model sizes and code tasks.

Critical strengths
  • Most existing RLVR methods focus on getting better rollouts from a fixed set of problems. CLPO instead tries to change the training problems based on the model's current ability. I think this is an interesting direction, as the training data can evolve together with the model.
  • I also like the idea of learning how to rewrite problems, rather than simply using fixed prompts to generate more data. CLPO uses the accuracy difference before and after rewriting as feedback, so the model can learn to generate more useful training problems. The joint training of solving and rewriting is also a reasonable design. The ablation in Table 3 shows that removing the rewriting objective leads to worse results on both math and code tasks.
  • CLPO shows clear improvements on Qwen3-8B, especially on harder math benchmarks such as Minerva Math and OlympiadBench. The authors also report improvements on MMLU-Pro, TheoremQA, and GPQA-Diamond, which suggests some generalization beyond math reasoning.
Critical weaknesses
  • My first concern is about the number of rollouts used to estimate problem difficulty. CLPO only uses G=4 rollouts to decide whether a problem is hard, medium, or easy. However, model generation is quite stochastic, especially with temperature=1.0. I feel that only four samples may not be enough to reliably estimate how difficult a problem actually is for the model. This could also affect the ΔAcc reward used to train the rewriting policy. Although Appendix B.3 discusses the extra cost of using more rollouts, there are no sensitivity experiments. I think this is an important hyperparameter, and it would be helpful to compare G=4, 8, and 16 in terms of difficulty estimation, final performance, and training cost, to better understand the trade-off.
  • Another concern is that CLPO does not actually handle the hardest problems where all rollouts fail. One of the main motivations is to simplify difficult problems so that the model can learn from them. However, Section 3.2 filters out all 0/4 problems before rewriting. So CLPO only simplifies problems that the model can already solve at least once out of four attempts. I think this is a bit different from the original motivation. It would be helpful if the authors could explain this design choice and explore whether simplifying these zero-success problems could also provide useful training signals.
  • I am also a bit concerned about the maximum response length used in the experiments. The authors use 8,192 tokens for both training and evaluation, while Qwen3-8B supports a much longer context length. For difficult math benchmarks like AIME and OlympiadBench, some problems may require longer reasoning. Although all baselines use the same token limit, different models may be affected differently by truncation. I did not find any analysis of the truncation rate or how performance changes with longer response lengths. I think it would be helpful to compare the results with larger token budgets, such as 16K or 32K, to see whether the improvements still hold.
Questions for authors
  • I noticed that Table 5 only compares CLPO with GRPO and DAPO, but does not include Critique-GRPO (CoT), which is the strongest baseline in Table 1. Could the authors also include this baseline in the fixed-time comparison?
  • I am also curious why AIME2025 and AIME2026 are included in Table 5 but not in the main results in Table 1. Could the authors explain this choice?
  • CLPO performs online problem rewriting at every training step, which introduces additional generation and evaluation costs. Although the authors provide wall-clock comparisons, I wonder if rewriting is really needed at every step. For example, could the authors use some cheaper signals, such as recent reward statistics or model uncertainty, to decide when a problem needs to be rewritten? It would be interesting to see whether similar performance can be achieved with less frequent rewriting and lower training costs.
  • One small writing issue: the citations around Lines 51 and 81 look a bit awkward. I think using \citep{} would be more appropriate here.
  • Another minor suggestion is that the benchmark citations in Lines 93–97 could be moved to the experimental setup section, where these benchmarks are introduced in more detail.
Reviewer_hR2tOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a variant of RL based on a curriculum learning framework that continuously identifies easy, medium, and hard problems. It simplifies the hard set and rewrites the medium ones to increase diversity. The model itself does the rewriting, and it gets rewarded by how much its accuracy improves on the rewritten version of a problem. The resulting hybrid set is used to train the LLM in the same update. They test this approach (called CLPO) mainly on math reasoning.

Critical strengths

I find the idea of having the policy rewrite its own training problems (simplifying hard ones and rephrasing medium ones) quite interesting, especially since the rewriting is rewarded by the accuracy gain it brings. The evaluation also covers a good number of baselines and benchmarks, including out-of-domain ones.

Critical weaknesses
  1. The main weakness is that the comparisons do not include any curriculum learning baselines. The paper only compares against SFT methods and RLVR algorithms that do not use a curriculum. There are several papers that propose ideas related to sampling problems from easy to hard, or sampling problems the current model fails at [1, 2, 3, 4]. The Kimi k1.5 tech report [5] also mentions curriculum sampling and prioritized sampling in its RL stage (Section 2.3.4). To establish the benefit of the proposed approach, it would be helpful to compare against some of these strategies. It is not clear to me whether gains come from the adaptive problem restructuring in Figure 2 or simply from having some adaptive difficulty-based curriculum (which could be replaced by one of the other ones already proposed in the literature).

  2. Two of the math benchmarks have fewer than 50 problems (AMC23 has 40 and AIME24 has 30), and Table 1 samples a single answer per question. I understand that these are established benchmarks for mathematical reasoning, but on these two, CLPO ties the strongest baseline on AMC23 and beats it on AIME24 by one problem only. With 32 samples per problem (Table 4), the gap is ~2 points on AIME24 and is slightly negative on AMC23. So even the paper's own repeated-sampling results show a pretty small gap. Due to the noise in LLM benchmarking [6], it is difficult to establish a real improvement on these benchmarks from single-sample results. I would expect to see multiple-sampling results that compare CLPO against the curriculum learning approaches from point 1, with the average and standard deviation across all problem sets. I do acknowledge that improvements on other datasets were more substantial.

[1]: Shi et al., Efficient Reinforcement Finetuning via Adaptive Curriculum Learning, TMLR 2026

[2]: Parashar et al., Curriculum Reinforcement Learning from Easy to Hard Tasks Improves LLM Reasoning, ICLR 2026

[3]: Chen et al., Self-Evolving Curriculum for LLM Reasoning, arXiv May 2025

[4]: Wang et al., DUMP: Automated Distribution-Level Curriculum Learning for RL-based LLM Post-training, arXiv April 2025

[5]: Kimi Team, Kimi k1.5: Scaling Reinforcement Learning with LLMs, arXiv January 2025

[6]: Hochlehnert et al., A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibility, COLM 2025

Questions for authors

How does CLPO compare to an adaptive curriculum alone without any problem rewriting (e.g. simply sampling problems by accuracy)? This would show some evidence that some of the gain comes from restructuring the problems (see weakness 1 above).

2 comments

Thank you for the thoughtful review and for pointing out the need for curriculum-only controls and a clearer account of uncertainty on the smaller math benchmarks.

Q1: How does CLPO compare with an adaptive curriculum that selects original problems without rewriting them?

A1: We evaluated history-prioritized sampling, a curriculum-only baseline that adaptively samples original problems based on the model’s performance, without using the problem-rewriting module. It follows the prioritized sampling rule described in Kimi k1.5 (Section 2.3.4) [1], assigning sampling weights proportional to one minus each problem’s historical success rate.

We trained the baseline from Qwen3-8B for 200 updates, matching the backbone and update count in the submitted paper, and evaluated it with thinking enabled. The results are shown below.

BenchmarkAccuracy metric (%)History-prioritized samplingCLPO
MATH500Pass@188.2090.80
AIME24Avg@3256.9853.13
AMC23Avg@3284.2285.08
Minerva-MathPass@145.9672.79
OlympiadBenchPass@166.0777.50
AVGUnweighted mean68.2975.86

Under their respective evaluation settings, the unweighted summary averages 75.86% for CLPO and 68.29% for history-prioritized sampling. CLPO is higher on four of five benchmarks; AIME24 is lower (53.13 vs. 56.98), indicating that problem restructuring makes a crucial contribution to CLPO’s gains.

Q2: Under repeated sampling, how does CLPO compare with curriculum learning baselines in mean accuracy and standard deviation on each benchmark?

A2: For the two smaller benchmarks, we additionally report repeated-sampling uncertainty under a shared protocol with 32 responses per problem.

BenchmarkCLPO Avg@32 (%) ± SD (pp)History-prioritized sampling Avg@32 (%) ± SD (pp)
AIME2453.13 ± 6.9856.98 ± 9.04
AMC2385.08 ± 4.0984.22 ± 5.75

CLPO has lower standard deviations on AIME24 and AMC23 than history-prioritized sampling. However, on AIME24 CLPO’s mean is lower (53.13 vs. 56.98). As shown in A1, CLPO’s reported advantage over history-prioritized sampling is mainly on MATH500, Minerva-Math, and OlympiadBench.

[1] Kimi Team. Kimi k1.5: Scaling Reinforcement Learning with LLMs.

Reviewer_hR2tOpenReview ↗

response to authors

I appreciate the additional experiments. But the comparison does not seem to use the same evaluation settings for both methods, and on the benchmarks where it does (AIME24 and AMC23), the curriculum-only baseline is either better or within noise. So if anything, these results suggest that most of the gains come from the curriculum, not from rewriting the problems. I also still don’t see comparisons to the other curriculum learning approaches I mentioned, or repeated-sampling results with the standard deviation across all benchmarks (not only the two small ones). Due to the remaining concern, I am inclined to keep my original score of weak reject.

Reviewer_5NvmOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CLPO lets the problem set move with the policy. Within each update, prompts that are neither already solved nor hopeless are split by group accuracy into hard, medium and easy; the policy itself then rewrites the hard ones to be simpler and the medium ones to be reworded, and the rewrites are solved with fresh rollouts. The rewriting is not treated as data augmentation: each rewriting trajectory is scored by how much accuracy rose on its own rewrite, and is optimized in the same update as the solving trajectories at a tenth of the weight. Results are on Qwen3-8B over five math and three out-of-domain benchmarks, averaging 72.25 against 62.04 for GRPO and 64.50 for DAPO, which is where the abstract's +10.21 and +7.75 come from; the strongest baseline in the same table sits at 66.77. The paper also reports per-checkpoint curves, Avg@32 on two competition benchmarks, ablations of the two rewriting modes and of the rewriting loss on math and code, and a comparison at fixed wall clock.

Critical strengths
  1. The margins sit on Minerva-Math and OlympiadBench, which are large enough that a few lucky items cannot explain an 11 to 14 point gap.

  2. Table 5 compares at fixed wall clock, which is the right answer to the obvious objection that a rewriting pipeline buys accuracy with extra generation.

Critical weaknesses
  1. The signal driving the method is a difference of two four-sample estimates, and it pays most for trivialization. App. H says a rewrite whose group comes out all-correct gets positive feedback - the largest signal available on a hard problem - while one that leaves the problem unsolvable loses only 0.25 to 0.50, less than the -1 for a malformed output. Semantics are never checked in the loop, the faithfulness audit uses Qwen3-Max rather than the trained policy, and the count of valid rewrites is never reported, so how often the shortcut was taken is unmeasured.

  2. Matched on steps, not on generation. Baselines produce 256 solving responses per step while CLPO adds 33 rewriting trajectories plus re-solving on top, roughly half again as much early on, and V is never reported. Table 5 is the only measured cost and it omits Critique-GRPO (CoT), the strongest baseline at 66.77, against which the margin is 5.48 rather than the advertised 10.21.

Questions for authors
  1. What is V per step, and what are CLPO's total responses and tokens at 200 steps against GRPO and DAPO? Does the Table 1 margin hold at matched generation?
  2. What fraction of positively rewarded rewrites ended all-correct, i.e. were made trivial rather than clarified?
  3. What is Critique-GRPO (CoT) at your 10-hour and 20-hour budgets?
  4. What is the clipping range of the surrogate objective, and is the Qwen3-8B starting point the base or the instruction-tuned checkpoint?
  5. Can you give three seeds with intervals for Tables 2-4, and state what decontamination was done between the DAPO-Math pool, its rewritten variants, and the math benchmarks?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.