acceptodds
Under review as a conference paper at ICLR 2027

Learning to Self-Train: Recursive Self-Improvement for Coding Agents

Abstract

Learning to self-train requires a coding agent to diagnose failures, construct useful practice, and solve the resulting tasks. We introduce AI4AI, a teacher-assisted framework that trains these behaviors together through recursive supervised fine-tuning. A single student model serves in three prompt-conditioned roles: a Critic diagnoses capability gaps from execution traces, a Producer constructs repository-grounded practice tasks, and a Solver generates candidate repairs. Execution-based admission checks filter the generated supervision, and joint supervised fine-tuning updates all three roles. Successive updates therefore change both the agent and the supervision it helps generate. Teacher demonstrations initialize the process, followed by increasing student contributions within each role. Across eight training requests starting from a 4B model, the 500-task resolution rate rises from 7.2% at the original base model to **20.2% after cold start and 62.2% at the final request**, a gain of **42.0 percentage points** beyond cold start. In controlled role evaluations, Critic diagnosis accuracy rises from **39.0% to 78.5%** on 200 held-out failure traces, and Producer valid-task yield rises from **7.0% to 31.5%** over 200 proposals per model. With the R8 Solver fixed, R8 Critic feedback raises retry success from **19.0% without feedback to 33.5%** on 200 failed tasks. The results show improvements in coding, diagnosis, and task generation, with Critic feedback providing a measurable benefit to repair.

Primary area:
foundation or frontier models, including LLMs
Keywords:
recursive self-improvement, self-training, coding agents, synthetic data generation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_wAn1OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces AI4AI framework that improves a target LLM's coding capability with the three roles namely Critic, Producer, and Solver. According to the authors, AI4AI starts with synthetic data generated by a larger teacher LLM, which accounts for 100% of the training data for the first three rounds. Then, the target LLM learns to generate its own data to improve its coding capability, reducing the portion of teacher-generated data. The effectiveness of the AI4AI framework is evaluated by showing that the resulting model performs better than the best-performing baseline (Agent-R, 55.0%) by 1.2 percentage points (56.2% vs. 55.0%) on 500 coding tasks.

Critical strengths

The paper proposes an interesting framework that separates recursive self-training into three roles, namely Critic, Producer, and Solver. This provides a clear structure for failure diagnosis, task generation, and problem solving.

Critical weaknesses

The paper presents an interesting framework, but the experimental results do not sufficiently validate its key contributions or explain the factors behind the reported improvements.

1. Insufficient evidence of recursive self-improvement

The AI4AI framework heavily relies on teacher-generated data, even after cold start. In R8, 50% of the Solver training data still comes from the teacher LLM. However, the paper does not provide experiments without continued teacher assistance or a teacher-only baseline. Therefore, it is unclear whether the performance improvement comes from the student's self-improvement or continued teacher supervision.

2. Insufficient validation of the proposed three-role framework

The main contribution of this paper is the framework consisting of Critic, Producer, and Solver. However, the paper does not provide ablation studies to show whether all three roles are necessary. Although the proposed method outperforms the existing baselines, it is unclear whether the improvement actually comes from the proposed three-role design.

3. Insufficient analysis of agent and feedback dependency

The experiments are conducted using a single agent framework, OpenHands. However, the paper does not investigate how the agent framework or the information contained in execution traces affects the self-training process. It is unclear which feedback information contributes to the improvement and whether the proposed framework would work similarly with different agent frameworks.

4. Limited generalizability across LLMs

The experiments use only one teacher LLM (deepseek-ai/deepseek-v4-flash-0731) and one student LLM (Qwen3.5-4B-Base). However, the paper does not investigate whether the proposed framework works with different LLMs or model sizes. Therefore, it is unclear whether the reported improvements can be generalized to other teacher and student models.

Questions for authors

I would appreciate clarification on the following points, as additional evidence could affect my assessment.

  • Is there evidence that the student LLM can continue improving without teacher assistance after cold start? Are there any results comparing AI4AI with a teacher-only baseline under a similar training budget?
  • Are there any ablation results showing that all three roles are necessary? In particular, how much does the Critic contribute compared to using only Producer and Solver?
  • Is there any analysis showing which information in the execution traces contributes to self-improvement? Is there any evidence that the improvements are not specific to OpenHands?
  • Are there any additional results with different teacher or student LLMs, or different model sizes, that support the generalizability of the proposed framework?
  • (Minor) Was there a particular reason for referring to the arXiv versions of these papers rather than their published conference versions?
Reviewer_6KG9OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This manuscript proposes AI4AI, a teacher-assisted training framework in which one coding model serves three roles: diagnosing failures, constructing repository-based practice tasks, and solving those tasks. Accepted outputs from all three roles supervise subsequent model updates, with student-generated records progressively replacing teacher-generated supervision. Across the eight training rounds, the reported task-resolution rate increases from 20.2% after initialization to 56.2%, compared with 55.0% for an adapted Agent-R baseline. Additional discussion experiments measure diagnosis quality, task-generation validity, and feedback usefulness. The contribution is a coding agent that improves both its repair ability and its ability to produce subsequent training data.

Critical strengths
  1. The controlled role evaluations provide meaningful evidence of useful capabilities. Diagnosis evaluation uses the same 200 excluded failure traces and blinded human assessment across checkpoints. The feedback experiment holds the Solver and retry budgets fixed. This may support the practical value of the learned Critic.
  2. The task-generation design includes a concrete validity check. Requiring tests to pass before mutation, fail after mutation, and pass after restoration helps establish that a generated defect causes the intended failure. These choices strengthen the evidence that the model learns to construct usable practice tasks.
Critical weaknesses
  1. The main performance evaluation is insufficiently specified to establish independent generalization or a reliable advantage over baselines. Section 4.2 identifies the benchmark only as a “500-task evaluation,” without specifying its dataset, version, task selection, or repository composition. Appendix B describes requirements for isolation without documenting their application to the reported experiment. Thus, the reviewers cannot determine whether the headline results measure generalization or performance on tasks that influenced training and selection. Meanwhile, the six-task advantage over Agent-R comes without paired outcomes or uncertainty estimates.
  2. The experiments do not isolate the contribution of recursive student-generated training. During the R1 to R3, performance already rises from 20.2% to 49.4%. The final model adds 6.8 percentage points beyond R3. Teacher supervision continues throughout, including half of the R8 Solver records. Meanwhile, training-set size changes substantially across rounds, and later token accounting remains unresolved (Tables 9–12). The role evaluations demonstrate improved capabilities, and the retry experiment demonstrates feedback utility, but neither establishes that the proposed recursive mechanism improves subsequent training outcomes.
Questions for authors
  1. What exactly are the 500 evaluation tasks, and were any of their identities, traces, outcomes, or aggregate scores used for diagnosis, task generation, hyperparameter decisions, or checkpoint selection? What separation and overlap checks were actually applied?
  2. How were the comparison methods implemented and matched for training exposure, teacher assistance, and inference budgets? What are the paired wins and losses against Agent-R, and what uncertainty accompanies the reported difference?
  3. How do the R6 and R8 performance declines relate to those adoption decisions?
  4. Under matched training and data-generation budgets, does updating the student Critic and Producer outperform continued teacher-only supervision or keeping those roles fixed? Can an ablation establish that diagnosis-conditioned practice improves downstream coding performance?
Reviewer_yPWWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

AI4AI is utilizing one shared model in three roles; (i) Critic diagnoses execution failures, (ii) Producer generates repository-grounded practice tasks, and Solver repairs them; (iii) execution-filtered records from all three activities were used for iterative supervised fine-tuning. Student-generated records are gradually substituted for those created under teacher supervision. The final 4B model is resolving 56.2% of 500 tasks, compared to 20.2% after initialization and 55.0% for the adapted Agent-R baseline. Separate experiments evaluate diagnosis quality, task-generation validity, and feedback-assisted recovery. The main question is whether learning these three roles results in a more effective training curriculum than continued training alone would achieve.

Critical strengths
  • The component evaluations test additional capabilities beyond the final benchmarks' accuracy. The models diagnose the same 200 held-out traces under blinded human assessment. The producer evaluations consider all proposals, including failed ones. The diagnosis accuracy increases from 39.0% to 78.5%, and the valid-task yield increases from 7.0% to 31.5%.
  • The fixed-Solver experiment demonstrates that feedback can improve recovery. By fixing the failure states, the Solver, and the retry limit, we find that R8 Critic feedback repairs 67/200 traces, compared to 52/200 for Self-Refine-style feedback and 38/200 for no feedback.
Critical weaknesses
  • The framework distinguishes the adaptive feedback set F and optional held-out set H, but does not clarify which set contains the reported 500-task evaluation. It is unclear whether these tasks, their traces, or their scores influenced diagnosis, generation, or checkpoint selection.
  • The advantage over the strongest baseline is six tasks (281 vs. 275). The paper provides neither paired wins/losses nor variability across training runs. Implementation and resource-matching details for the adapted baselines are lacking, and the robustness of this small advantage.
  • Of the 36.0-point improvement after initialization, 29.2 occurs during R1–R3, wherein all training records are teacher-generated, even at R8. The missing info is a budget-matched, teacher-only continuation that measures what student participation adds.
  • The retry experiment tests the utility of feedback at inference-time. It does not test if Critic-guided generation improves further training.
  • Equation 2's check shows that a mutation changed the focused test’s outcome. It does not establish that the diagnosis correctly identifies the capability responsible for the original failure, nor that practicing the generated task improves that capability.
Questions for authors
  • What are the 500 tasks, and which of the supplied datasets provided adaptive feedback and development selection?
  • Do matched teacher-only continuation and frozen/omitted Critic–Producer training controls retain the gains? Does diagnosed-gap conditioning outperform shuffled or absent gap conditioning at comparable generation and training budgets?
  • Which components of each baseline were retained, and how were teacher access, training tokens, and inference budgets controlled?
  • What are the actual role prompts, diagnosis-admission rules, protected repair checks, gap-allocation policy, and checkpoint-adoption criterion?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.