- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how to make RLVR training data better match the model's current reasoning ability. The authors point out that most RLVR methods train on a fixed set of problems. However, as the model improves, some problems may become too easy or remain too hard, and thus provide little useful learning signal. To address this, the authors propose CLPO, which dynamically adjusts training problems based on the model's rollout accuracy. It first divides problems into hard, medium, and easy groups. Hard problems are simplified, while medium problems are diversified. The authors also introduce a restructuring-aware objective, which uses the accuracy difference before and after rewriting (ΔAcc) to reward useful rewrites. The model is then jointly trained on original problems, rewritten problems, and rewriting trajectories. Experiments on Qwen3-8B show that CLPO achieves an average accuracy of 72.25% across eight benchmarks, outperforming GRPO, DAPO, and Critique-GRPO (CoT) by 10.21, 7.75, and 5.48 points, respectively. The authors also provide ablations, fixed-time training comparisons, and experiments on different model sizes and code tasks.
Critical strengths
- Most existing RLVR methods focus on getting better rollouts from a fixed set of problems. CLPO instead tries to change the training problems based on the model's current ability. I think this is an interesting direction, as the training data can evolve together with the model.
- I also like the idea of learning how to rewrite problems, rather than simply using fixed prompts to generate more data. CLPO uses the accuracy difference before and after rewriting as feedback, so the model can learn to generate more useful training problems. The joint training of solving and rewriting is also a reasonable design. The ablation in Table 3 shows that removing the rewriting objective leads to worse results on both math and code tasks.
- CLPO shows clear improvements on Qwen3-8B, especially on harder math benchmarks such as Minerva Math and OlympiadBench. The authors also report improvements on MMLU-Pro, TheoremQA, and GPQA-Diamond, which suggests some generalization beyond math reasoning.
Critical weaknesses
- My first concern is about the number of rollouts used to estimate problem difficulty. CLPO only uses G=4 rollouts to decide whether a problem is hard, medium, or easy. However, model generation is quite stochastic, especially with temperature=1.0. I feel that only four samples may not be enough to reliably estimate how difficult a problem actually is for the model. This could also affect the ΔAcc reward used to train the rewriting policy. Although Appendix B.3 discusses the extra cost of using more rollouts, there are no sensitivity experiments. I think this is an important hyperparameter, and it would be helpful to compare G=4, 8, and 16 in terms of difficulty estimation, final performance, and training cost, to better understand the trade-off.
- Another concern is that CLPO does not actually handle the hardest problems where all rollouts fail. One of the main motivations is to simplify difficult problems so that the model can learn from them. However, Section 3.2 filters out all 0/4 problems before rewriting. So CLPO only simplifies problems that the model can already solve at least once out of four attempts. I think this is a bit different from the original motivation. It would be helpful if the authors could explain this design choice and explore whether simplifying these zero-success problems could also provide useful training signals.
- I am also a bit concerned about the maximum response length used in the experiments. The authors use 8,192 tokens for both training and evaluation, while Qwen3-8B supports a much longer context length. For difficult math benchmarks like AIME and OlympiadBench, some problems may require longer reasoning. Although all baselines use the same token limit, different models may be affected differently by truncation. I did not find any analysis of the truncation rate or how performance changes with longer response lengths. I think it would be helpful to compare the results with larger token budgets, such as 16K or 32K, to see whether the improvements still hold.
Questions for authors
- I noticed that Table 5 only compares CLPO with GRPO and DAPO, but does not include Critique-GRPO (CoT), which is the strongest baseline in Table 1. Could the authors also include this baseline in the fixed-time comparison?
- I am also curious why AIME2025 and AIME2026 are included in Table 5 but not in the main results in Table 1. Could the authors explain this choice?
- CLPO performs online problem rewriting at every training step, which introduces additional generation and evaluation costs. Although the authors provide wall-clock comparisons, I wonder if rewriting is really needed at every step. For example, could the authors use some cheaper signals, such as recent reward statistics or model uncertainty, to decide when a problem needs to be rewritten? It would be interesting to see whether similar performance can be achieved with less frequent rewriting and lower training costs.
- One small writing issue: the citations around Lines 51 and 81 look a bit awkward. I think using \citep{} would be more appropriate here.
- Another minor suggestion is that the benchmark citations in Lines 93–97 could be moved to the experimental setup section, where these benchmarks are introduced in more detail.
Thank you for the thoughtful review and for pointing out the need for curriculum-only controls and a clearer account of uncertainty on the smaller math benchmarks.
A1: We evaluated history-prioritized sampling, a curriculum-only baseline that adaptively samples original problems based on the model’s performance, without using the problem-rewriting module. It follows the prioritized sampling rule described in Kimi k1.5 (Section 2.3.4) [1], assigning sampling weights proportional to one minus each problem’s historical success rate.
We trained the baseline from Qwen3-8B for 200 updates, matching the backbone and update count in the submitted paper, and evaluated it with thinking enabled. The results are shown below.
Under their respective evaluation settings, the unweighted summary averages 75.86% for CLPO and 68.29% for history-prioritized sampling. CLPO is higher on four of five benchmarks; AIME24 is lower (53.13 vs. 56.98), indicating that problem restructuring makes a crucial contribution to CLPO’s gains.
A2: For the two smaller benchmarks, we additionally report repeated-sampling uncertainty under a shared protocol with 32 responses per problem.
CLPO has lower standard deviations on AIME24 and AMC23 than history-prioritized sampling. However, on AIME24 CLPO’s mean is lower (53.13 vs. 56.98). As shown in A1, CLPO’s reported advantage over history-prioritized sampling is mainly on MATH500, Minerva-Math, and OlympiadBench.
[1] Kimi Team. Kimi k1.5: Scaling Reinforcement Learning with LLMs.