acceptodds
Under review as a conference paper at ICLR 2027

Learning at and Beyond the Competence Boundary for On-Policy Post-Training

Abstract

On-policy post-training adapts learning to a model's current behavior, while adaptive curriculum learning adjusts task selection to its evolving competence. Yet adapting responses and selecting from existing tasks still leaves a critical gap: persistent failures reveal capability deficits but do not by themselves create the new practice needed to overcome them. To address this gap, we propose CASE, a Competence-Aware Selection with Error-targeted generation framework for adaptive on-policy post-training. CASE adapts both the response and task distributions: at each round, it retains tasks near the student's current competence boundary and uses a teacher agent to diagnose recurring failures and generate verified intermediate problems targeting the underlying weaknesses. The student then learns from freshly sampled responses on the selected and generated tasks using group relative policy optimization with executable rewards. We evaluate CASE with Qwen-3.5-9B across code, math, and finance, including analyses of task selection, error-targeted generation, and teacher choice. The results show complementary gains from selection and generation, with an average pass@1 improvement of 7.69 percentage points over the base model on out-of-domain benchmarks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM post-training, curriculum learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_v6wDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CASE is a multi-round on-policy post-training framework that keeps the GRPO objective unchanged and adapts only the task distribution. Each round, it: (i) retains tasks with mixed rollout outcomes; (ii) uses a teacher agent to turn validation failures into error reports and generate execution-verified new problems; (iii) carries forward or rolls back checkpoints based on validation scores.

With Qwen-3.5-9B trained on APPS, the paper reports an average +7.69 pass@1 improvement on 9 OOD benchmarks.

Critical strengths

The generation pipeline is carefully engineered: diagnosis and authoring are separated, and each problem is verified with the training grader. This is useful for reproducibility, but it is an engineering rather than a methodological contribution.

Critical Weaknesses

Limited novelty and generality. The method combines established components: mixed-outcome filtering is essentially DAPO’s dynamic sampling, applied offline; failure-driven task generation has direct precedents (e.g., Sentinel, cited but not compared); execution-verified synthetic problems are a standard data recipe. Moreover, the selection criterion is derived from the zero-advantage property of GRPO under binary rewards. The method is only validated with GRPO on single-turn code tasks, so it is unclear whether it transfers to other RL algorithms or settings. Insufficient and unsupportive ablations. Table 4 does not support the claimed complementary gains: adding targeted generation lowers MATH500 (90.40 → 88.76) and HMMT25 (60.00 → 56.67); untargeted generation beats targeted generation on MATH500; DAPO outperforms CASE on MATH500 and AIME24. There is no cost-matched baseline, even though CASE uses frontier-model API calls every round. The validation sets are also reused for diagnosis, checkpoint selection, and reporting. Finally, some statistics are impossible: AIME24 (30 problems, 5 runs) is reported as 63.33 ± 0.43 in Table A.5.

Critical weaknesses

The generation pipeline is carefully engineered: diagnosis and authoring are separated, and each problem is verified with the training grader. This is useful for reproducibility, but it is an engineering rather than a methodological contribution.

Critical Weaknesses

  1. Limited novelty and generality. The method combines established components: mixed-outcome filtering is essentially DAPO’s dynamic sampling, applied offline; failure-driven task generation has direct precedents (e.g., Sentinel, cited but not compared); execution-verified synthetic problems are a standard data recipe. Moreover, the selection criterion is derived from the zero-advantage property of GRPO under binary rewards. The method is only validated with GRPO on single-turn code tasks, so it is unclear whether it transfers to other RL algorithms or settings.

  2. Insufficient and unsupportive ablations. Table 4 does not support the claimed complementary gains: adding targeted generation lowers MATH500 (90.40 → 88.76) and HMMT25 (60.00 → 56.67); untargeted generation beats targeted generation on MATH500; DAPO outperforms CASE on MATH500 and AIME24.

  3. There is no cost-matched baseline, even though CASE uses frontier-model API calls every round. The validation sets are also reused for diagnosis, checkpoint selection, and reporting. Finally, some statistics are impossible: AIME24 (30 problems, 5 runs) is reported as 63.33 ± 0.43 in Table A.5.

Questions for authors

Can CASE be extended to other RL algorithms, at least REINFORCE and PPO, and to agentic RL (multi-turn, tool-use settings)? How would the selection criterion be defined when advantages come from a learned critic or when rewards are not binary?

Beyond offline filtering, what distinguishes boundary selection from DAPO’s dynamic sampling? Can you provide a cost-matched baseline, e.g., the same teacher budget spent on untargeted data or on SFT from teacher solutions?

Please explain why targeted generation underperforms on the math benchmarks, and correct the impossible statistics in Tables 4 and A.5.

6 comments

W1: Limited Novelty

We thank the reviewer for the thoughtful comments. While CASE builds on established techniques, its main contribution lies in integrating competence-aware selection and error-targeted generation into a closed-loop curriculum that continuously adapts to the student's evolving capabilities. Unlike DAPO's optimization-time dynamic sampling or Sentinel's failure-driven task generation, CASE assigns distinct roles to different task outcomes. Mixed-outcome tasks support direct optimization, and persistent failures guide targeted curriculum expansion. Generated tasks enter a cumulative pool and are reassessed in subsequent rounds, allowing their roles to evolve with student competence. Thus, CASE goes beyond combining existing components by establishing a feedback loop between competence assessment, task selection, targeted generation, and on-policy optimization. Table 4 further supports the benefits of this joint adaptation over dynamic sampling and untargeted generation baselines.

Following the suggestions from Reviewer sdMr and your valuable feedback, we have conducted additional comparisons with three closely related approaches: SOAR, SPADE, and Sentinel. Since these methods were originally developed for different settings, we adapted their core mechanisms to executable code reasoning. SOAR uses student learning progress on difficult problems to guide task generation. SPADE employs hint-based regret for environment generation, and Sentinel generates tasks based on diagnosed student failures. All methods use Qwen-3.5-9B as the initial student and APPS-train as the initial training dataset, with matched student optimization budgets, task verification procedures, and evaluation protocols. We report pass@1 on the same nine OOD benchmarks. Due to time constraints, the current results are based on a single experimental run. We will further conduct five independent runs to assess the robustness of the findings and revise the final results in revision later.

CodeContestGSM8KMATH500AIME25AIME24HMMT26HMMT25FinReasonFinChain
base30.9468.7386.1256.6763.3332.7346.6784.2380.58
SOAR39.1371.6590.4063.3360.0036.3646.6786.3382.31
SPADE41.3068.6188.2046.6753.3336.3643.3384.3480.21
Sentinel39.4969.6088.6046.6760.0042.4243.3384.0481.48
CASE45.3674.7788.7664.6772.0046.6756.6787.3082.52

The performance differences may reflect how these methods construct and adapt their curricula. SOAR selects generated tasks based on student progress on a designated set of difficult problems, potentially favoring localized improvements over broader generalization. SPADE uses hint-based regret to prioritize learnable tasks, while its environment-design mechanisms may be less effective in single-turn code reasoning. Sentinel generates tasks from diagnosed failures but lacks explicit competence-based reassessment of the accumulated task pool, potentially limiting training efficiency as student competence evolves.

W2: Insufficient and Unsupportive Ablations

We thank the reviewer for the careful assessment. We agree that the observed improvements are not uniform across benchmarks and appreciate the opportunity to clarify the evidence for the complementary contributions of competence-aware selection and error-targeted generation.

1. Performance drops after adding Targeted Generation. We acknowledge the decreases on MATH500 and HMMT25. Nevertheless, adding Targeted Generation to Boundary Selection improves performance on 10 of the 12 benchmarks, with an average pass@1 gain of 4.10 pp. Since the training and generated tasks are code-based, their benefits may transfer differently to OOD mathematics and finance benchmarks. We therefore interpret the results as evidence of an overall benefit from combining the two components, while recognizing that the gains are task-dependent.

2. Targeted vs. Untargeted Generation. Unlike Untargeted Generation, which follows general task-generation instructions, Targeted Generation uses diagnosed student failures to construct practice addressing specific capability gaps. To better isolate its contribution, we conducted four additional analyses on 1,000 problems from each strategy, using GPT-5.6 Sol as the teacher:

  • Error Alignment: Using the 116 error clusters identified in the diagnostic reports as references, Targeted Generation covers 90.52% of the clusters, compared with 68.97% for Untargeted Generation. Moreover, 93.70% of targeted problems align with diagnosed error clusters, versus 65.50% of untargeted problems.
  • Category Diversity: We group generated problems into 21 algorithmic categories and quantify their diversity using the exponential of Shannon entropy, Deff=exp⁡(−∑ipilog⁡pi)D_{\mathrm{eff}}=\exp(-\sum_i p_i\log p_i), where pip_i denotes the proportion of problems in category ii. This metric represents the effective number of equally represented categories, accounting for both category coverage and distributional balance. Targeted Generation achieves 19.2 effective categories, compared with 16.3 for Untargeted Generation, and consistently maintains higher diversity across all five rounds.
  • Task Difficulty and Test Coverage: An independent evaluation using DeepSeek-R1 finds that 40.7% of targeted problems are medium or hard, compared with 27.6% of untargeted problems. Targeted problems also contain more test cases on average, 7.9 per problem versus 3.5, with 2.2% of test inputs meeting the stress-test threshold, compared with none in the untargeted set.
  • Within-category Similarity: Targeted Generation yields lower text similarity in 20 of the 21 algorithmic categories, with a weighted average of 0.120, compared with 0.145 for Untargeted Generation.

These analyses provide further evidence for the contribution of Error-targeted Generation beyond simply adding synthetic training data. Consistent with these findings, Targeted Generation outperforms Untargeted Generation on 11 of the 12 benchmarks in Table 4, under the same Boundary Selection strategy and generated task volume.

3. Comparison with DAPO. We recognize that DAPO achieves better results on some mathematics benchmarks, including MATH500. However, under the same Error-targeted Generation setting, CASE performs better on 11 of the 12 benchmarks spanning code, math, and finance. This broader comparison suggests that round-level competence-aware selection can provide additional benefits beyond DAPO's dynamic sampling, although the relative advantage varies across tasks.

Overall, we agree that the results should not be interpreted as uniform improvements across all benchmarks. We will revise the discussion to distinguish aggregate gains from benchmark-specific variations and incorporate the additional analyses to better substantiate the contribution of Error-targeted Generation.

W3: Computational Cost, Validation Reuse, and Statistical Reliability

Regarding Cost-matched Baselines. We thank the reviewer for raising this important concern. We would like to clarify that our ablation experiments already control the major factors affecting training fairness. Specifically, Targeted and Untargeted Generation use the same teacher model, generate an equal number of verified problems per round, and follow identical student models, verification procedures, and GRPO training configurations. Thus, their performance differences cannot be attributed simply to a larger generated dataset or additional student optimization steps.

We further provide a breakdown of the teacher API overhead. Using GPT-5.6 Sol, each round generates 200 verified problems through approximately 55-64 API calls, consuming an average of 3.29M input tokens and 72K output tokens. Under the stated API pricing and caching assumptions, this corresponds to an estimated generation cost of \3.4−5.1perround,or3.4-5.1 per round, or \\17-26 across five rounds. We will clarify this experimental protocol and report the teacher-side overhead in the revised manuscript.

Regarding Validation Reuse. We appreciate the reviewer's concern and would like to clarify two important safeguards in our evaluation protocol. First, CASE explicitly separates error diagnosis from task generation. The generation agent has no access to the original validation problems or student failure trajectories. Instead, it receives only sanitized, capability-level diagnostic reports summarizing recurring error patterns. This design prevents direct reformulation of validation instances during task generation and reduces the risk of data leakage.

Second, the three validation benchmarks are used for diagnosis and checkpoint selection, we additionally evaluate CASE on nine independent OOD test benchmarks spanning code, mathematics, and finance, none of which participate in training, task generation, or checkpoint selection. Therefore, we distinguish validation performance, which guides curriculum adaptation, from independent test performance, which provides the primary evidence of generalization. We will make this distinction clearer in the revised manuscript.

Regarding the Statistical Inconsistency in AIME24. We thank the reviewer for identifying this issue. The inconsistency is a typo when reporting the std result. We have carefully re-examined the original evaluation records and will correct the relevant statistics in the revised manuscript to ensure accuracy and reliability.

Reviewer_DdFWOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CASE, an adaptive curriculum framework for on-policy post-training. After an initial GRPO warm-up on APPS, the student filters the accumulated task pool using rollout outcomes and retains mixed-outcome tasks. A teacher separately diagnoses errors on three code validation benchmarks and generates new executable problems from category-level reports. The selected and generated tasks support fresh GRPO rollouts, while validation performance controls checkpoint carryover and rollback. Experiments with Qwen-3.5-9B and three teachers report improvements across nine out-of-distribution benchmarks in code, mathematics, and finance, with comparisons of task filtering, targeted generation, and DAPO.

Critical strengths

A sensible approach to curriculum adaptation. Combining the selection of tasks that provide reward contrast with the generation of new practice is well motivated. Separating failure diagnosis from isolated problem authoring is also a reasonable design for reducing direct reformulation of validation examples.

Substantial empirical evaluation with useful controls. The study includes three teachers, nine test benchmarks excluded from the adaptation loop, and comparisons with boundary-only training, equal-count untargeted generation, and DAPO. Appendix A.2 reports five independent training seeds. The full boundary-selection variants improve over the base model on every OOD benchmark in mean performance, providing credible evidence of transfer beyond the feedback benchmarks.

Critical weaknesses

The incremental advantage over strong alternatives needs stronger evidence. The relevant comparisons in Table 4 are more modest than the improvement over the base model. For the GPT-teacher configuration, the macro-average advantage across the nine OOD benchmarks is approximately 0.69 percentage points over DAPO without generation and 1.45 points over GRPO with boundary selection and untargeted generation. Table A.5 provides useful variability estimates for CASE variants, but not for these key controls. Moreover, adding targeted generation to DAPO lowers its OOD average by approximately 1.93 points relative to untargeted generation, requiring a more qualified interpretation of complementarity with adaptive sampling. The budget comparison also remains unclear: each adaptive round samples eight responses for every accumulated task before collecting fresh training responses. Reporting uncertainty for the key differences and matching total sampling budgets would better establish the value of the additional curriculum machinery.

The role of error alignment and learnable intermediate practice is not sufficiently established. The equal-count untargeted control is useful, but Figure 6 contrasts a simple Fibonacci-style problem with a more difficult permutation-statistic DP problem. Without characterizing the generated pools, improvements may reflect differences in difficulty or topic composition as well as alignment with current errors. Generated tasks are added after executable verification without a reported student-learnability criterion (Section 4.1). Appendix B shows plausible diagnosis-to-topic mappings, but does not establish that the new tasks form learnable intermediate steps toward resolving persistent failures. Student success rates on generated tasks, changes in the targeted error clusters, and a difficulty-matched or shuffled-report control would strengthen this central interpretation.

The reliability of the generated training rewards needs independent assessment. Appendix B.6 carefully checks executability and consistency with the training grader. However, for stdin and functional tasks, expected outputs are obtained by executing the same reference solution that is subsequently checked against them. This does not independently establish agreement with the problem statement. Test adequacy is also unclear: the inversion-DP example permits n up to 200 and k up to 20,000, while the four displayed inputs have n at most 6 (Appendix B.5.1). These inputs cannot test the claimed need for an efficient transition. An independent correctness audit and clearer test-coverage evidence would help assess the quality of the rewards used for GRPO.

Questions for authors

1 Could the authors provide variability for all Table 4 controls, uncertainty on the main differences, and total student-rollout and teacher-generation costs? How should the weaker DAPO results with targeted generation be reconciled with the claimed complementarity?

2 What are the student’s initial and subsequent success rates on generated tasks? Could a difficulty-matched or shuffled-report control, together with outcomes on held-out tasks from the diagnosed error clusters, distinguish error alignment from broader changes in the generated distribution?

3 Could the authors independently audit a random sample of generated statements, reference solutions, and tests, and report the observed error rate? Are the small input sets in Appendix B.5 complete test suites or illustrative subsets?

4 Please specify evaluation decoding and sample counts, the rollback tolerance and final checkpoint-selection rule, and whether the 1,000 APPS items in the Appendix B.4 diagnostic report are a subset of the 5,000 validation problems in Table 1.

I find the framework promising and appreciate the experimental effort. I would be open to reconsidering the current rating if the comparative evidence and generated-task reliability concerns are satisfactorily addressed.

7 comments

Q1 & W1: Performance, Cost, and Reliability

We thank the reviewer for raising important questions about performance improvements, computational costs, and statistical reliability.

Performance against Strong Baselines

We believe the two strong baselines in Table 4, DAPO without generation and GRPO with Boundary Selection and Untargeted Generation, help assess the incremental contributions of CASE from different perspectives. First, DAPO already employs dynamic sampling to filter informative training examples, using a criterion similar to our Boundary Selection. Comparing CASE against DAPO therefore examines whether incorporating Error-targeted Generation and curriculum adaptation provides additional benefits beyond adaptive sampling alone. Second, GRPO with Boundary Selection and Untargeted Generation uses the same task selection strategy, teacher model, and number of generated problems as CASE, allowing us to isolate the contribution of error alignment under comparable settings. Although CASE achieves relatively modest average gains, these improvements remain meaningful given that both baselines already incorporate effective adaptive sampling or task generation mechanisms. Moreover, CASE shows broadly positive results across 9 OOD benchmarks and consistently improves upon the base model under all 3 teacher configurations, suggesting robustness.

About statistical reliability, we will extend Table A.5 in the revised manuscript to include additional key baselines and report their mean and variant results for a more comprehensive comparison.

The performance decline observed when adding Targeted Generation to DAPO may be related to differences in how the two methods adapt their training-task distributions. DAPO applies dynamic sampling during optimization, filtering prompts without within-group reward variation to improve gradient utilization within each training batch. In contrast, CASE constructs an active curriculum based on the student's competence at the beginning of each round and explicitly incorporates generated tasks into the training pool. Consequently, even with identical generated problems, the two methods may encounter different effective training distributions. For example, targeted problems that are too difficult for the current student may be repeatedly filtered or resampled under DAPO, whereas CASE trains on a curriculum combining selected boundary tasks with newly generated problems. This observation suggests that optimization-time dynamic sampling alone may not adequately account for the student's longer-term capability gaps, potentially limiting its ability to benefit from error-targeted generation. We present this as a possible explanation rather than an experimentally established cause.

Cost and Sampling Budget

We would first like to distinguish three stages of CASE, competence estimation, training optimization, and teacher generation. Competence estimation is an inference-only procedure that assesses the student's competence boundary and constructs the active curriculum, without gradient computation or parameter updates. In contrast, GRPO training requires fresh on-policy rollouts, reward computation, and gradient-based optimization. Therefore, our experiments control the training optimization budget, rather than the total sampling compute. We also keep the growth of the cumulative task pool consistent across settings by adding 200 generated problems per round, resulting in comparable competence-estimation sampling requirements. Specifically, competence estimation samples eight responses per task across five adaptive rounds, yielding a total of 201,960 student rollouts, calculated as 8 × (4,649 + 4,849 + 5,049 + 5,249 + 5,449). GRPO training spans six rounds, each consisting of 15 optimization steps, 64 prompts per step, and eight responses per prompt, resulting in 46,080 training rollouts in total.

For teacher generation, CASE invokes frontier-model APIs through the SDK Agent to generate and self-verify new problems. Taking the Error-targeted Generation setting with GPT-5.6 Sol as an example, generating 200 verified problems per round requires approximately 55-64 API calls for batched generation and self-verification. Based on actual API token usage records, each round consumes an average of approximately 3.29 million input tokens and 72,000 output tokens. The substantially larger input-token count arises from repeated context transmission during multi-turn agent interactions, including error diagnosis reports, generation instructions, and previously generated problems.

Using API rates of \4permillioninputtokens,4 per million input tokens, \\0.40 per million cached input tokens, and \20permillionoutputtokens,theestimatedgenerationcostisapproximately20 per million output tokens, the estimated generation cost is approximately \\3.4-5.1 per round, assuming an 80%-95% prompt-cache hit rate. Across five generation rounds producing 1,000 verified problems, this amounts to approximately \17-26 in total, or \0.02-0.03 per problem.

W2: Error Alignment and Generated Task Characterization

We thank the reviewer for the suggestions regarding error alignment and generated-task learnability. We agree that case studies alone are insufficient to rule out the effects of task difficulty and topic distribution. Therefore, we provide a more fine-grained characterization of the generated task pools along four dimensions: error alignment, category diversity, task difficulty, and within-category similarity. All results reported below use GPT-5.6 Sol as the teacher model.

Analysis 1: Error Alignment

Unlike Targeted Generation, which first produces a detailed error report identifying the student's failure types and error clusters, Untargeted Generation has no access to either the error report or the failed problems. We therefore compare the 1,000 tasks generated by each setting against the 116 error clusters identified in the targeted setting, using the same reference clusters for both methods.

MetricTargeted GenerationUntargeted Generation
# Generated Tasks1,0001,000
# Diagnosed Error Clusters116-- (116 targeted-setting clusters used as reference)
Error Cluster Coverage105 (90.52%)80 (68.97%)
Tasks Matching Diagnosed Clusters937 (93.70%)655 (65.50%)

Targeted Generation achieves substantially higher error-cluster coverage than Untargeted Generation (90.52% vs. 68.97%). Moreover, 93.70% of targeted tasks match the diagnosed error clusters, compared with 65.50% of untargeted tasks. These results confirm that Targeted Generation produces tasks more closely aligned with the student's diagnosed weaknesses, whereas Untargeted Generation lacks access to the error information needed for such alignment.

Analysis 2: Category Diversity

To examine whether Targeted Generation produces an overly narrow task distribution, we group the 116 diagnosed error clusters into 21 broader algorithmic categories, including dynamic programming, graph and tree algorithms, and greedy algorithms. We then classify the generated tasks into these categories to compare their topic distributions.

We measure diversity, defined as the exponential of Shannon entropy Deff=exp⁡(H),H=−∑i=121piln⁡piD_{\mathrm{eff}}=\exp(H),H=-\sum_{i=1}^{21}p_i\ln p_i, where pip_i denotes the proportion of generated tasks belonging to category ii. A higher value indicates greater topic diversity.

setroundnn# categories usedDiversityLargest category
Targeted12002117.8greedy & tree algorithm 10%
Targeted22002016.6PANDAS operations 12%
Targeted32001816.2Bitwise/XOR 14%
Targeted42002016.6Numpy/Scipy operations 12%
Targeted52002117.9greedy & tree algorithm 11%
Targetedall10002119.2greedy & tree algorithm 10%
Untargeted12001712.3Numpy/Scipy operations 9%
Untargeted22001913.7greedy algorithm 16%
Untargeted32001710.6PANDAS operations 29%
Untargeted42001713.6Range-query Data Structure 17%
Untargeted5200139.3PLOT 31%
Untargetedall10002116.3PANDAS operation 14%

Targeted Generation consistently achieves higher category diversity across all five adaptive rounds. Its dominant categories also vary across rounds, without excessive concentration on any single category. Overall, Targeted Generation achieves a higher diversity than Untargeted Generation, indicating a more balanced topic distribution. These results suggest that targeting diagnosed errors does not compromise category-level diversity, but instead maintains a broad and balanced task distribution.

Targeted Generation consistently exhibits higher category diversity than Untargeted Generation across all five adaptive rounds, with varying dominant categories and no excessive concentration in any single category. Its higher overall diversity further indicates a broader and more balanced topic distribution, demonstrating that targeting diagnosed errors does not come at the expense of category-level diversity.

AuthorsOpenReview ↗

Continue answering the previous question

Analysis 3: Task Difficulty and Test Coverage

To objectively compare task difficulty, we use an independent third-party model, DeepSeek-R1, to assign difficulty labels to both generated task pools, rather than relying solely on the teacher's self-assessments. Among the 1,000 targeted tasks, 59.3% are easy, 37.6% medium, and 3.1% hard, compared with 72.4%, 27.4%, and 0.2%, respectively, for Untargeted Generation. Thus, Targeted Generation produces a higher proportion of medium- and high-difficulty tasks.

We further examine test coverage. Untargeted Generation provides an average of 3.5 test cases per algorithmic task, with at most five tests per task. In contrast, Targeted Generation provides 7.9 test cases per task on average, with 2.2% of test inputs reaching stress-test scale. For reference, APPS and LCB contain an average of 7.8 and 11.7 test cases per task, respectively. These results indicate that Targeted Generation produces more challenging tasks with broader test coverage, including some large-scale inputs.

Analysis 4: Within-category Similarity

Beyond category-level diversity, we examine whether tasks within the same algorithmic category are overly similar. Using the 21 categories defined above, we represent each problem statement as a TF-IDF vector after masking numerical values. We then compute pairwise cosine similarity s(xi,xj)=vi⊤vj∥vi∥∥vj∥s(x_i,x_j)=\frac{\mathbf v_i^\top\mathbf v_j}{\|\mathbf v_i\|\|\mathbf v_j\|}, and average it across all task pairs within each category Sc=2nc(nc−1)∑i<js(xi,xj)S_c=\frac{2}{n_c(n_c-1)}\sum_{i<j}s(x_i,x_j), where ncn_c is the number of tasks in category cc.

We compare Targeted Generation, Untargeted Generation, and Benchmark Failures (problems the student failed on the original benchmarks). Lower similarity indicates less lexical repetition within a category.

CategoryTargeted Generation TasksUntargeted Generation TasksBenchmark Failures
Counting/comb. mod p0.132 (n=65)0.141 (n=76)0.138 (n=119)
Optimization DP0.088 (n=30)0.117 (n=45)0.107 (n=136)
Digit DP0.241 (n=28)0.337 (n=7)0.134 (n=13)
Graph & Tree0.103 (n=95)0.131 (n=82)0.121 (n=147)
Greedy/Binary Search0.076 (n=70)0.100 (n=76)0.111 (n=206)
Range-query DS0.103 (n=29)0.126 (n=67)0.150 (n=84)
Number Theory0.111 (n=22)0.140 (n=60)0.121 (n=101)
Bitwise/XOR0.123 (n=28)0.154 (n=36)0.157 (n=32)
Strings0.078 (n=16)0.180 (n=24)0.118 (n=107)
Geometry/Float0.103 (n=21)0.135 (n=23)0.091 (n=55)
Game Theory0.187 (n=25)0.456 (n=19)0.172 (n=24)
Constructive/Ad-hoc0.075 (n=16)0.117 (n=63)0.092 (n=218)
Matplotlib/Seaborn0.147 (n=56)0.181 (n=62)0.078 (n=276)
Pandas Operations0.129 (n=69)0.147 (n=136)0.081 (n=168)
Scikit-learn0.118 (n=27)0.310 (n=7)0.084 (n=55)
NumPy/SciPy/Stats0.128 (n=69)0.106 (n=56)0.077 (n=59)
File/OS/Subprocess0.127 (n=36)0.130 (n=45)0.105 (n=85)
Regex/Text/Collections0.074 (n=43)0.124 (n=54)0.084 (n=47)
Datetime/Timezone0.145 (n=26)0.249 (n=15)0.065 (n=41)
Serialization/Hash/HTTP0.126 (n=52)0.131 (n=34)0.073 (n=109)
Pure-Python Contract0.153 (n=5)0.156 (n=13)—
Weighted Mean0.1200.1450.102

Targeted Generation exhibits lower within-category similarity than Untargeted Generation in 20 of the 21 categories, with a weighted mean of 0.120 vs. 0.145, approaching the 0.102 observed for Benchmark Failures. These results suggest that targeting specific student weaknesses does not necessarily lead to repetitive problem statements. Instead, Targeted Generation maintains substantial lexical diversity within algorithmic categories while preserving its focus on diagnosed errors.

Reviewer_sdMrOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Summary: This paper proposes a curriculum generation scheme for RL post training, which both selects problems to fit the agent's current zone of proximal development and generates new problems to fill the gaps between existing problems and the agent's current capabilities.

Critical strengths

This work is a solid contribution to automated curricula design for RL post-training. It provides a scaled up implementation of many ideas from the field of automatic curriculum generation in RL and figures out many of the important details necessary to make these techniques work for RL post-training. The empirical results seem solid, and the methods are well-motivated.

There are several aspects of this work which are truly novel. For instance, I have not seen this sort of agentic curricula design in prior work, the closest would be OMNI-EPIC[6] or SPADE[7], but these do not give the task designer as much freedom. Moreover, there are not many examples of using these sorts of curricula generation approaches in RL post training, I believe every work in this space provides extremely valuable signal to the field, and this work in particular is valuable in the way it is well scoped and evaluated. I believe that with proper contextualisation within prior work, this would be a strong contribution to the field.

Critical weaknesses

My main concern with this work comes from how it positions itself with respect to prior work and how it compares against prior work. While it cites the prior work in curricula learning for ML, the more direct prior work is that of the curricula learning in RL. Specifically, many aspects of this work have direct analogs in the well-developed field of unsupervised environment design. For instance, the buffer-based competence selection criteria appears similar to the prioritised level replay[1] using a priorization rule like the one proposed in sampling for learnability[2]. The promotion rule for students in closed-loop adaptation is similar to the criteria used in POET[3], and the one used in SOAR[4] for keeping improvements to students. The error-targeted task generation in equation 8 where the task is proposed with a reference solution is a version of the 2-player PAIRED[5] game where the adversary and the antagonist (who solves the adversaries problems to check they are solvable), are the same player. The combination of the generative and curation approaches was studied in DCD[6], with the combination of PAIRED and PLR specifically studied.

More generally, from reading the introduction and abstract, a reader could get the impression that no one has generated (rather than curated) curricula for RL systems before, but this is quite a large field of study which is still being actively built on. For broader reference see POET[3], Enhanced POET[10], PCGviaRL[7], PAIRED[5], PLR[1], ADA[8], SFL[2], DRED[9], CLUTR[11], ACCEL[12]

Ideally, there would be a comparison the these existing approaches for generating curricula, the two most appropriate comparisons here would be SOAR and SPADE. At minimum, it would be important to discuss these works and describe the comparative benefits to your approach.

[1]Jiang, Minqi, Edward Grefenstette, and Tim Rocktäschel. "Prioritized level replay." International Conference on Machine Learning. PMLR, 2021.

[2]Foster, Thomas, et al. "Learning to reason at the frontier of learnability." arXiv preprint arXiv:2502.12272 (2025).

[3] Wang, Rui, et al. "Paired open-ended trailblazer (poet): Endlessly generating increasingly complex and diverse learning environments and their solutions." arXiv preprint arXiv:1901.01753 (2019).

[4] Sundaram, Shobhita, et al. "Teaching models to teach themselves: Reasoning at the edge of learnability." arXiv preprint arXiv:2601.18778 (2026).

[5] Dennis, Michael, et al. "Emergent complexity and zero-shot transfer via unsupervised environment design." Advances in neural information processing systems 33 (2020): 13049-13061.

[6] Jiang, Minqi, et al. "Replay-guided adversarial environment design." Advances in Neural Information Processing Systems 34 (2021): 1884-1897.

[7] Khalifa, Ahmed, et al. "Pcgrl: Procedural content generation via reinforcement learning." Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment. Vol. 16. No. 1. 2020.

[8] Team, Adaptive Agent, et al. "Human-timescale adaptation in an open-ended task space." arXiv preprint arXiv:2301.07608 (2023).

[9] Garcin, Samuel, et al. "DRED: Zero-shot transfer in reinforcement learning via data-regularised environment design." arXiv preprint arXiv:2402.03479 (2024).

[10] Wang, Rui, et al. "Enhanced poet: Open-ended reinforcement learning through unbounded invention of learning challenges and their solutions." International conference on machine learning. PMLR, 2020.

[11] Azad, Abdus Salam, et al. "Clutr: Curriculum learning via unsupervised task representation learning." International Conference on Machine Learning. PMLR, 2023.

[12] Parker-Holder, Jack, et al. "Evolving curricula with regret-based environment design." International Conference on Machine Learning. PMLR, 2022.

Questions for authors

Could you provide interesting and typical examples of the curricula tasks you have learned? Often the qualitative behaviours of these systems are a clear sign of how they are working, and given how unique your setup is, the tasks that it generates may be quite interesting.

5 comments

W1: Positioning with Respect to Prior Curriculum Learning and Unsupervised Environment Design

We sincerely thank the Reviewer for recognizing the value of our work and for providing detailed and thoughtful suggestions on the relevant research literature. We agree that the current version does not sufficiently discuss prior work on curriculum learning in RL, particularly unsupervised environment design (UED), leaving the connections and distinctions between CASE and existing approaches insufficiently articulated. We will address this limitation and clarify our research positioning in the revision.

We would also like to take this opportunity to further explain the original motivation behind this paper. Our original intention was not to introduce a new curriculum learning paradigm, but rather to explore how to build more effective on-policy post-training frameworks that enable LLMs to learn autonomously and continuously improve their capabilities. Inspired by auto-research, OPD, and a growing body of recent self-learning research, we began to consider a fundamental question. Rather than relying solely on a model's own feedback, could guidance from a stronger teacher model, capable of analyzing the student's current learning state and underlying causes of failure, help the student more effectively overcome its existing competence boundary?

This question also reminded us of an intuitive learning strategy in traditional education that students need not only to reinforce what they have already learned through practice, but also to analyze recurring mistakes and develop the ability to generalize from them by solving new problems that exercise similar knowledge and skills without simply repeating the original questions. We believe this learning process offers valuable insights into improving LLM reasoning capabilities. In particular, in on-policy RL, a model's current performance can reveal both which tasks still provide effective optimization signals and which knowledge or reasoning patterns it has not yet mastered. We therefore sought to explore whether these two sources of information could be leveraged together. Selecting tasks that match the student's current competence while using teacher model to transform persistent failures into new, targeted learning opportunities.

Motivated by this perspective, we developed CASE, which integrates competence-aware selection and error-targeted Generation into a unified closed-loop post-training framework. We acknowledge the Reviewer's observation that these mechanisms share important conceptual connections with prior UED research, including prioritized level replay, learning at the frontier of learnability, environment generation, and student promotion. We do not claim to be the first to introduce task selection, generative curricula, or closed-loop model selection. Instead, CASE focuses on integrating student competence estimation, teacher-guided failure diagnosis, and targeted task generation within LLM on-policy post-training, allowing the training curriculum to continuously adapt to the student's evolving competence. Specifically, the teacher model in CASE does not directly provide target answers for the student to imitate. Instead, it analyzes recurring capability gaps revealed by the student's failures and generates new, executable-verified practice problems targeting these weaknesses. The student continues to learn from its own on-policy responses and verifiable rewards through GRPO optimization. The central question we aim to investigate is, beyond selecting learnable tasks from an existing pool, can a stronger teacher identify the underlying causes of student failures and proactively construct targeted new problems to further improve LLM reasoning capabilities? We believe this represents the specific research direction explored by CASE while building upon established ideas in curriculum learning.

We appreciate the Reviewer's references to PLR, SFL, POET, PAIRED, DCD, SOAR, SPADE, and the broader UED literature. These studies are highly relevant to understanding the methodological design and research positioning of CASE. In the revised manuscript, we will expand the Related Work section and revise the Introduction and Abstract accordingly. Furthermore, we provide additional experimental comparisons with SOAR and SPADE, as well as Sentinel [13], a closely related work already cited in our manuscript but not previously included in our experimental comparisons.

[13] Sentinel: Failure-driven reinforcement learning for training tool-using language model agents.

AuthorsOpenReview ↗

Continue answering the previous question

Since these methods were originally developed for different task settings, we adapt them to the reasoning setting considered in our work while preserving their core mechanisms as much as possible. Specifically, for SOAR, we use GPT-5.6 Sol as the teacher and Qwen-3.5-9B as the student, consistent with our experimental setup. Since teacher model is accessed through the black-box API, we cannot directly optimize the teacher using RLOO as in the original SOAR framework. We therefore adapt SOAR by retaining its student-improvement-based curriculum evaluation and student promotion mechanisms while removing the gradient-based teacher update. Specifically, the teacher generates candidate programming tasks, and the student is trained on these tasks using RLOO in the inner loop. For SPADE, we use the teacher model as the environment designer, representing code reasoning tasks as single-step executable environments. The designer generates executable programming problems and corresponding hints, while hint-based regret guides curriculum selection and the solver is optimized using GRPO. For Sentinel, we adapt its original tool-use failure analysis to identify error patterns in code reasoning, where the controller diagnoses the student's algorithmic and implementation errors, and the proposer generates targeted executable programming tasks. We also retain its original task-buffer and GRPO training mechanisms.

To ensure a consistent comparison, all methods use Qwen-3.5-9B as the initial student model and APPS-train as the sole initial training dataset. We report pass@1 on the same nine OOD benchmarks. Due to time constraints, the current results are based on a single experimental run. We plan to conduct five independent runs and report the complete results in the revised manuscript. Additionally, as complete source code is not publicly available for some methods, we have reproduced these approaches based on our understanding of their published descriptions. We acknowledge that these independent implementations may not fully reproduce the performance of the original methods.

MethodCodeContestGSM8KMATH500AIME25AIME24HMMT26HMMT25FinReasonFinChain
base30.9468.7386.1256.6763.3332.7346.6784.2380.58
SOAR39.1371.6590.4063.3360.0036.3646.6786.3382.31
SPADE41.3068.6188.2046.6753.3336.3643.3384.3480.21
Sentinel39.4969.6088.6046.6760.0042.4243.3384.0481.48
CASE45.3674.7788.7664.6772.0046.6756.6787.3082.52

The observed performance differences may be attributed to the curriculum construction mechanisms and optimization objectives of these methods. SOAR uses student learning progress on a designated set of difficult problems to select generated tasks, which may favor improvements in specific reasoning patterns rather than broader generalization. SPADE primarily relies on hint-based regret to guide environment generation, emphasizing the construction of appropriately learnable tasks rather than directly optimizing their contribution to downstream reasoning performance. Moreover, its original design targets more complex interactive environments, and its advantages may not be fully realized in the single-turn code reasoning setting. Sentinel, while generating targeted tasks based on the student's observed failure patterns, primarily focuses on failure-driven curriculum expansion without an explicit mechanism for continuously matching the accumulated task pool to the student's evolving competence. Consequently, its limited training budget may not always be concentrated on the most informative tasks.

In conclusion, we sincerely thank the reviewer again for these valuable references and constructive suggestions. Your feedback has helped us recognize the limitations of the current manuscript in terms of research positioning and literature coverage, and will help us present our contributions more accurately and comprehensively in the revised version.

Q1: Case Studies

Thank the Reviewer for this valuable suggestion. We agree that qualitative examples are particularly useful for understanding how an adaptive curriculum responds to the student's evolving capabilities. In fact, we have already included several representative examples in Figure 6 and Appendix B.4 and B.5, covering both the generated problems and the underlying failure diagnoses that motivate them. For instance, Appendix B.4 presents a failure-analysis report from round 3, where the teacher identifies persistent weaknesses in counting DP state design and implicit graph modeling, as well as an emerging weakness in computational geometry. Based on these diagnoses, Appendix B.5 illustrates how the teacher generates targeted practice problems. For counting DP, the teacher generates a permutation-inversion counting problem that requires the student to introduce an appropriate DP state and optimize its transitions, directly addressing the diagnosed state-design weakness. For implicit graph modeling, the generated problem requires recognizing arithmetic transitions as edges in an implicit graph before applying shortest-path search. For computational geometry, the teacher generates a circle-intersection problem that exercises geometric case analysis and numerical precision, targeting a weakness that becomes more prominent as the student improves.

These examples illustrate that CASE does not simply generate harder problems or reformulate previously failed questions. Instead, it identifies recurring reasoning deficiencies and constructs new tasks that exercise the underlying skills through different problem formulations. We appreciate the Reviewer's suggestion and will make these existing examples more prominent in the revised manuscript, with clearer explanations connecting the diagnosed failures, generated tasks, and their intended learning objectives.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global Response

We sincerely thank the reviewers and the AC for your constructive feedback. In the revised manuscript, we have made several substantive changes. In the main paper, we added the new ablation suggested by Reviewer DdFW, which randomly shuffles the assignment of diagnostic reports and their associated task batches across training rounds to evaluate the importance of temporal alignment (Table 4). We also added the comparison methods requested by Reviewers sdMr and v6wD (Table 5) and, following Reviewer sdMr’s suggestion, expanded the Related Work section to discuss a broader range of relevant studies. In the appendix, we corrected and substantially expanded Table A.6 (formerly Table A.5), and added Table A.7 to report additional important baseline variants based on GRPO, DAPO, and related configurations. We further added a representative generated problem with numerous test cases covering boundary and extreme conditions in Section B.5.5. Finally, Section B.7 now provides a fine-grained analysis of generated-task quality, including error alignment, category diversity, task difficulty and test coverage, and within-category similarity.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.