- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
CASE is a multi-round on-policy post-training framework that keeps the GRPO objective unchanged and adapts only the task distribution. Each round, it: (i) retains tasks with mixed rollout outcomes; (ii) uses a teacher agent to turn validation failures into error reports and generate execution-verified new problems; (iii) carries forward or rolls back checkpoints based on validation scores.
With Qwen-3.5-9B trained on APPS, the paper reports an average +7.69 pass@1 improvement on 9 OOD benchmarks.
Critical strengths
The generation pipeline is carefully engineered: diagnosis and authoring are separated, and each problem is verified with the training grader. This is useful for reproducibility, but it is an engineering rather than a methodological contribution.
Critical Weaknesses
Limited novelty and generality. The method combines established components: mixed-outcome filtering is essentially DAPO’s dynamic sampling, applied offline; failure-driven task generation has direct precedents (e.g., Sentinel, cited but not compared); execution-verified synthetic problems are a standard data recipe. Moreover, the selection criterion is derived from the zero-advantage property of GRPO under binary rewards. The method is only validated with GRPO on single-turn code tasks, so it is unclear whether it transfers to other RL algorithms or settings. Insufficient and unsupportive ablations. Table 4 does not support the claimed complementary gains: adding targeted generation lowers MATH500 (90.40 → 88.76) and HMMT25 (60.00 → 56.67); untargeted generation beats targeted generation on MATH500; DAPO outperforms CASE on MATH500 and AIME24. There is no cost-matched baseline, even though CASE uses frontier-model API calls every round. The validation sets are also reused for diagnosis, checkpoint selection, and reporting. Finally, some statistics are impossible: AIME24 (30 problems, 5 runs) is reported as 63.33 ± 0.43 in Table A.5.
Critical weaknesses
The generation pipeline is carefully engineered: diagnosis and authoring are separated, and each problem is verified with the training grader. This is useful for reproducibility, but it is an engineering rather than a methodological contribution.
Critical Weaknesses
-
Limited novelty and generality. The method combines established components: mixed-outcome filtering is essentially DAPO’s dynamic sampling, applied offline; failure-driven task generation has direct precedents (e.g., Sentinel, cited but not compared); execution-verified synthetic problems are a standard data recipe. Moreover, the selection criterion is derived from the zero-advantage property of GRPO under binary rewards. The method is only validated with GRPO on single-turn code tasks, so it is unclear whether it transfers to other RL algorithms or settings.
-
Insufficient and unsupportive ablations. Table 4 does not support the claimed complementary gains: adding targeted generation lowers MATH500 (90.40 → 88.76) and HMMT25 (60.00 → 56.67); untargeted generation beats targeted generation on MATH500; DAPO outperforms CASE on MATH500 and AIME24.
-
There is no cost-matched baseline, even though CASE uses frontier-model API calls every round. The validation sets are also reused for diagnosis, checkpoint selection, and reporting. Finally, some statistics are impossible: AIME24 (30 problems, 5 runs) is reported as 63.33 ± 0.43 in Table A.5.
Questions for authors
Can CASE be extended to other RL algorithms, at least REINFORCE and PPO, and to agentic RL (multi-turn, tool-use settings)? How would the selection criterion be defined when advantages come from a learned critic or when rewards are not binary?
Beyond offline filtering, what distinguishes boundary selection from DAPO’s dynamic sampling? Can you provide a cost-matched baseline, e.g., the same teacher budget spent on untargeted data or on SFT from teacher solutions?
Please explain why targeted generation underperforms on the math benchmarks, and correct the impossible statistics in Tables 4 and A.5.
6 comments
W2: Insufficient and Unsupportive Ablations
We thank the reviewer for the careful assessment. We agree that the observed improvements are not uniform across benchmarks and appreciate the opportunity to clarify the evidence for the complementary contributions of competence-aware selection and error-targeted generation.
1. Performance drops after adding Targeted Generation. We acknowledge the decreases on MATH500 and HMMT25. Nevertheless, adding Targeted Generation to Boundary Selection improves performance on 10 of the 12 benchmarks, with an average pass@1 gain of 4.10 pp. Since the training and generated tasks are code-based, their benefits may transfer differently to OOD mathematics and finance benchmarks. We therefore interpret the results as evidence of an overall benefit from combining the two components, while recognizing that the gains are task-dependent.
2. Targeted vs. Untargeted Generation. Unlike Untargeted Generation, which follows general task-generation instructions, Targeted Generation uses diagnosed student failures to construct practice addressing specific capability gaps. To better isolate its contribution, we conducted four additional analyses on 1,000 problems from each strategy, using GPT-5.6 Sol as the teacher:
- Error Alignment: Using the 116 error clusters identified in the diagnostic reports as references, Targeted Generation covers 90.52% of the clusters, compared with 68.97% for Untargeted Generation. Moreover, 93.70% of targeted problems align with diagnosed error clusters, versus 65.50% of untargeted problems.
- Category Diversity: We group generated problems into 21 algorithmic categories and quantify their diversity using the exponential of Shannon entropy, , where denotes the proportion of problems in category . This metric represents the effective number of equally represented categories, accounting for both category coverage and distributional balance. Targeted Generation achieves 19.2 effective categories, compared with 16.3 for Untargeted Generation, and consistently maintains higher diversity across all five rounds.
- Task Difficulty and Test Coverage: An independent evaluation using DeepSeek-R1 finds that 40.7% of targeted problems are medium or hard, compared with 27.6% of untargeted problems. Targeted problems also contain more test cases on average, 7.9 per problem versus 3.5, with 2.2% of test inputs meeting the stress-test threshold, compared with none in the untargeted set.
- Within-category Similarity: Targeted Generation yields lower text similarity in 20 of the 21 algorithmic categories, with a weighted average of 0.120, compared with 0.145 for Untargeted Generation.
These analyses provide further evidence for the contribution of Error-targeted Generation beyond simply adding synthetic training data. Consistent with these findings, Targeted Generation outperforms Untargeted Generation on 11 of the 12 benchmarks in Table 4, under the same Boundary Selection strategy and generated task volume.
3. Comparison with DAPO. We recognize that DAPO achieves better results on some mathematics benchmarks, including MATH500. However, under the same Error-targeted Generation setting, CASE performs better on 11 of the 12 benchmarks spanning code, math, and finance. This broader comparison suggests that round-level competence-aware selection can provide additional benefits beyond DAPO's dynamic sampling, although the relative advantage varies across tasks.
Overall, we agree that the results should not be interpreted as uniform improvements across all benchmarks. We will revise the discussion to distinguish aggregate gains from benchmark-specific variations and incorporate the additional analyses to better substantiate the contribution of Error-targeted Generation.
W3: Computational Cost, Validation Reuse, and Statistical Reliability
Regarding Cost-matched Baselines. We thank the reviewer for raising this important concern. We would like to clarify that our ablation experiments already control the major factors affecting training fairness. Specifically, Targeted and Untargeted Generation use the same teacher model, generate an equal number of verified problems per round, and follow identical student models, verification procedures, and GRPO training configurations. Thus, their performance differences cannot be attributed simply to a larger generated dataset or additional student optimization steps.
We further provide a breakdown of the teacher API overhead. Using GPT-5.6 Sol, each round generates 200 verified problems through approximately 55-64 API calls, consuming an average of 3.29M input tokens and 72K output tokens. Under the stated API pricing and caching assumptions, this corresponds to an estimated generation cost of \17-26 across five rounds. We will clarify this experimental protocol and report the teacher-side overhead in the revised manuscript.
Regarding Validation Reuse. We appreciate the reviewer's concern and would like to clarify two important safeguards in our evaluation protocol. First, CASE explicitly separates error diagnosis from task generation. The generation agent has no access to the original validation problems or student failure trajectories. Instead, it receives only sanitized, capability-level diagnostic reports summarizing recurring error patterns. This design prevents direct reformulation of validation instances during task generation and reduces the risk of data leakage.
Second, the three validation benchmarks are used for diagnosis and checkpoint selection, we additionally evaluate CASE on nine independent OOD test benchmarks spanning code, mathematics, and finance, none of which participate in training, task generation, or checkpoint selection. Therefore, we distinguish validation performance, which guides curriculum adaptation, from independent test performance, which provides the primary evidence of generalization. We will make this distinction clearer in the revised manuscript.
Regarding the Statistical Inconsistency in AIME24. We thank the reviewer for identifying this issue. The inconsistency is a typo when reporting the std result. We have carefully re-examined the original evaluation records and will correct the relevant statistics in the revised manuscript to ensure accuracy and reliability.
W1: Limited Novelty
We thank the reviewer for the thoughtful comments. While CASE builds on established techniques, its main contribution lies in integrating competence-aware selection and error-targeted generation into a closed-loop curriculum that continuously adapts to the student's evolving capabilities. Unlike DAPO's optimization-time dynamic sampling or Sentinel's failure-driven task generation, CASE assigns distinct roles to different task outcomes. Mixed-outcome tasks support direct optimization, and persistent failures guide targeted curriculum expansion. Generated tasks enter a cumulative pool and are reassessed in subsequent rounds, allowing their roles to evolve with student competence. Thus, CASE goes beyond combining existing components by establishing a feedback loop between competence assessment, task selection, targeted generation, and on-policy optimization. Table 4 further supports the benefits of this joint adaptation over dynamic sampling and untargeted generation baselines.
Following the suggestions from Reviewer sdMr and your valuable feedback, we have conducted additional comparisons with three closely related approaches: SOAR, SPADE, and Sentinel. Since these methods were originally developed for different settings, we adapted their core mechanisms to executable code reasoning. SOAR uses student learning progress on difficult problems to guide task generation. SPADE employs hint-based regret for environment generation, and Sentinel generates tasks based on diagnosed student failures. All methods use Qwen-3.5-9B as the initial student and APPS-train as the initial training dataset, with matched student optimization budgets, task verification procedures, and evaluation protocols. We report pass@1 on the same nine OOD benchmarks. Due to time constraints, the current results are based on a single experimental run. We will further conduct five independent runs to assess the robustness of the findings and revise the final results in revision later.
The performance differences may reflect how these methods construct and adapt their curricula. SOAR selects generated tasks based on student progress on a designated set of difficult problems, potentially favoring localized improvements over broader generalization. SPADE uses hint-based regret to prioritize learnable tasks, while its environment-design mechanisms may be less effective in single-turn code reasoning. Sentinel generates tasks from diagnosed failures but lacks explicit competence-based reassessment of the accumulated task pool, potentially limiting training efficiency as student competence evolves.