acceptodds
Under review as a conference paper at ICLR 2027

Recursive Self-Improvement through Defect-Driven Verifiable Environments

Abstract

Recursive self-improvement (RSI) requires a model to identify what to learn next and obtain reliable supervision for capabilities it has not yet mastered. We introduce a defect-driven RSI framework that turns recurring model failures into verifiable learning environments. We find that many failures are prompt-recoverable: the model fails unaided but succeeds with defect-specific structural guidance. This observation motivates using unaided performance and assisted verification success to target capabilities that remain deficient but are accessible under guidance. We use initialized executable environments to adapt training-problem distributions, generate and verify guided solution trajectories, and train the learner without guidance in its inputs. After each round, we reassess the updated learner to adjust data allocation and use it for controlled rewriting of subsequent problem descriptions, closing the loop between capability improvement and curriculum adaptation. In experiments with SIRL-Qwen2.5-7B on three optimization modeling benchmarks, we obtain relative pass@1 gains of 34.02% on MAMO ComplexLP and 12.62% on OptMATH-bench-193 over the base model. Across rounds, we observe an overall increase in the proportion of unaided successes, with defect-specific guidance, we obtain times as many verified trajectories as with unaided generation under the same sampling budget. Together, we use defect-specific structural guidance to link learning-target selection, dynamic capability assessment, and training-data generation, and we use the constructed environments to support execution and verification throughout this recursive improvement process.

Primary area:
foundation or frontier models, including LLMs
Keywords:
recursive self-improvement, verifiable environments, adaptive curriculum learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_wPa9OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a self-training framework for optimization modeling. The current model is evaluated with and without hints, and the resulting performance gap determines the allocation of subsequent training data. A fixed copy of the initial model generates hint-assisted solution trajectories, which are filtered by verifiers. The accepted solutions are then used for LoRA-based supervised fine-tuning with the hints removed from the training inputs. This assessment, allocation, and training cycle is repeated. The method is evaluated on three optimization-modeling benchmarks.

Critical strengths

None

Critical weaknesses
  1. The effectiveness of the performance-gap allocation metric is insufficiently established.

The comparison against uniform allocation provides useful evidence, but it does not establish whether the proposed gap offers information beyond ordinary problem difficulty. In particular, comparison with allocation based on the unassisted failure rate, expressed as 1 - p_plain, is needed to distinguish the value of hint-based assessment from simply prioritizing problems the model struggles to solve.

Moreover, the paper does not directly demonstrate that a larger performance gap predicts greater subsequent learning gains. This leaves the central interpretation of the allocation signal insufficiently validated.

  1. The contribution of recursive adaptation is not sufficiently isolated.

The experiments do not compare repeated reassessment against a curriculum determined by the initial assessment and then held fixed. Consequently, it remains unclear whether updating the allocation throughout training provides a meaningful advantage over a static curriculum.

Without this control, the benefits of recursive adaptation are difficult to distinguish from those of additional supervised fine-tuning on verified, externally guided solutions. A comparison using the same solution-generation procedure and matched training budgets would help establish the specific contribution of recursive reassessment.

Questions for authors
  1. Could the authors compare the proposed allocation rule against allocation based on the unassisted failure rate, 1 - p_plain, while keeping the solution-generation procedure, accepted-example budget, and training settings consistent?

  2. Could the authors provide paired statistical tests and confidence intervals for the improvements on NL4Opt and OptMATH-bench-193, both over the initial model and over uniform allocation? Since the reported variation reflects evaluation seeds, could the authors also assess variability across independent runs of the training pipeline?

3 comments
AuthorsOpenReview ↗

Response to Reviewer wPa9 (1/2): Additional experiments and statistical evidence

Overview of the additional experiments

Thank you for your specific and constructive comments on our work. To address these concerns, we added three controls with matched training budgets: allocation based on the frozen initial gap, dynamic failure-rate allocation, and offline SFT. All three additional experiments use 15 rounds, 128 training records per round, an effective batch size of 8, and 240 optimizer updates, with evaluation fixed at round 15 (R15). This matches the number of training examples and the update budget of the main experiment.

The central design of our method is to translate the learner's current capability deficiencies into subsequent training decisions. The assisted condition provides a way to assess how amenable failures are to intervention; round-by-round evaluation feeds the updated model state back into training allocation; and execution-based verification checks the outputs. The resulting recursive loop is "model state → deficiency measurement → training selection → model update → remeasurement." The additional controls examine different aspects of this design, rather than treating all components as a single factor.

The mean pass@1 scores (%) on NL4Opt, MAMO, and OptMATH are as follows:

ConditionNL4Opt245MAMO203OptMATH193
Frozen initial-gap allocation93.0670.6141.45
Dynamic failure-rate allocation93.0671.2638.51
Offline SFT93.3372.5840.24
Dynamic gap (ours)93.4773.0740.07

Dynamic gap achieves the highest mean accuracy among the four conditions on NL4Opt and MAMO. Compared with the frozen initial-gap condition, it improves performance by 0.41 and 2.46 percentage points, respectively. Compared with dynamic failure-rate allocation, it improves performance by 0.41, 1.81, and 1.56 percentage points across the three benchmarks, respectively.

Thank you for suggesting a failure-rate baseline, a frozen initial-allocation control, and predictive analyses. We added the corresponding controls and present descriptive results separately from statistical inference.

Response to the concern about paired statistical evidence and gains over uniform allocation

Thank you for recommending paired analyses at the problem level. We checked the archived outputs and performed 20,000 paired cluster bootstrap resamples, using problems as the resampling unit and retaining the three decoding outcomes for each problem under each of the two methods. We also conducted paired sign-flip tests.

On OptMATH-bench-193, the main model improves from 35.58% for the initial model with available paired outputs to 40.07%, a gain of 4.49 percentage points, with a 95% confidence interval of [0.35, 8.12] percentage points. On NL4Opt, the mean improvement is approximately 0.41 percentage points; the 95% problem-cluster bootstrap interval is approximately [0, 0.95] percentage points, and the two-sided paired sign-flip test yields p = 0.25.

Overall, our method improves mean performance over the initial model on both OptMATH and NL4Opt. The improvement on OptMATH is statistically supported, whereas NL4Opt shows a smaller numerical gain. Relative to uniform allocation, our method improves performance by 4.10, 0.69, and 0.14 percentage points on MAMO, OptMATH, and NL4Opt, respectively. These mean differences are consistently positive, but their magnitudes vary substantially across tasks, with the largest gain on MAMO. We therefore distinguish the capability improvements from training as a whole from the incremental benefits of dynamic task allocation; the significance of these gains is task-dependent.

AuthorsOpenReview ↗

Response to Reviewer wPa9 (2/2): Training variability and allocation controls

Response to the concern about variability across independent training runs

Thank you for highlighting the distinction between evaluation randomness and training randomness. The currently reported standard deviations are computed across different decoding seeds at the same trained checkpoint and reflect variability during evaluation. The paired analyses likewise characterize evaluation uncertainty at fixed checkpoints. We have not yet completed independent training replications in this round of revisions. We therefore limit our conclusions to the training runs evaluated and do not interpret these standard deviations as evidence of stability across independent training runs.

Response to the concern about failure-rate and frozen initial-gap controls

Dynamic failure-rate allocation selects samples using 1 - p_plain from its own model at each round, with the same exploration and capacity constraints as the original dynamic control. The frozen initial-gap condition retains the assistance-gap information from R0 but does not use post-training measurements to update priorities.

In the order NL4Opt / MAMO / OptMATH, the differences between dynamic gap and failure-rate allocation are +0.41 / +1.81 / +1.56 percentage points. These results show consistently positive differences in mean performance in favor of dynamic gap.

Thank you for the additional experiments.

However, the central interpretation of the performance gap remains insufficiently validated: the results do not directly establish that larger gaps predict greater subsequent learning gains. The allocation comparisons provide useful evidence, but do not fully justify this predictive interpretation.

I therefore maintain my rating.

Reviewer_76orOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a defect-driven Recursive self-improvement (RSI) framework that enables the model to learn from its failures by creating a corresponding training environment. The method utilizes defect-specific guidance to generate a verified solution trajectory and allocates training data according to the model’s both assisted and unassisted performance. The method is evaluated on three benchmarks: NL4Opt, MAMO ComplexLP, and OptMATH-bench-193 and gained 34.02% improvement on MAMO ComplexLP and 12.62% improvement on OptMath-bench-193. The authors also claim that by using the structural assistance, they can generate 6.49x more verified trajectories compared to the baseline given the same budget.

Critical strengths

For the strength of the paper,

  • The defect-driven verification environment generation provides a practical way to generate training environments given a recurring failure. This provides a useful and reliable supervision on the problems that the model cannot solve independently and enable the model to hillclimb them.
  • The empirical improvements are looking promising. The method shows a significant improvement on MAMO ComplexLP and also improves the efficiency of generating verified trajectories by 6.49x.
Critical weaknesses

For the weakness of the paper,

  • The paper claims to be a recursive self-improving framework; however, the environment generation and defect-specific guidance are based on using a stronger model with human review. Furthermore, the learner model only adjusts the curriculum and the problem description which is not a common way to claim it as a recursive “self” improvement.
  • While the paper evaluates adaptive v.s. uniform allocation, the improvement is marginal. The paper lacks the ablation isolating the contribution of each part of the method. Because of the lack of ablation it is hard to understand where the gains are actually coming from and how much the recursive mechanism contributes.
Questions for authors
  • The first thing that I want to ask is about the comparison against standard SFT using the same number of verified examples and fixed curriculum. Would removing the controlled rewriting affect the performance of the method significantly?

  • Can the authors explain more about why the method is recursive self improvement, not distillation from a better model?

2 comments
AuthorsOpenReview ↗

Response to Reviewer 76or

Overview of the additional experiments

Thank you for your specific and constructive comments on our work. To address these concerns, we added three controls with matched training budgets: allocation based on the frozen initial gap, dynamic failure-rate allocation, and offline SFT. All three additional experiments use 15 rounds, 128 training records per round, an effective batch size of 8, and 240 optimizer updates, with evaluation fixed at round 15 (R15). This matches the number of training examples and the update budget of the main experiment.

The central design of our method is to translate the learner's current capability deficiencies into subsequent training decisions. The assisted condition provides a way to assess how amenable failures are to intervention; round-by-round evaluation feeds the updated model state back into training allocation; and execution-based verification checks the outputs. The resulting recursive loop is "model state → deficiency measurement → training selection → model update → remeasurement." The additional controls examine different aspects of this design, rather than treating all components as a single factor.

The mean pass@1 scores (%) on NL4Opt, MAMO, and OptMATH are as follows:

ConditionNL4Opt245MAMO203OptMATH193
Frozen initial-gap allocation93.0670.6141.45
Dynamic failure-rate allocation93.0671.2638.51
Offline SFT93.3372.5840.24
Dynamic gap (ours)93.4773.0740.07

Dynamic gap achieves the highest mean accuracy among the four conditions on NL4Opt and MAMO. Compared with the frozen initial-gap condition, it improves performance by 0.41 and 2.46 percentage points, respectively. Compared with dynamic failure-rate allocation, it improves performance by 0.41, 1.81, and 1.56 percentage points across the three benchmarks, respectively.

Thank you for asking us to further distinguish recursive feedback, offline supervision, and external knowledge. Our response focuses on the dependencies within the learning process and the verifiable sources of supervision.

Response to the concern about fixed-curriculum offline SFT

The added offline SFT condition uses a fixed corpus and a globally shuffled training schedule, without reselecting samples based on current-model feedback during training. With matched sample counts and optimizer updates, it achieves 93.33 / 72.58 / 40.24, compared with 93.47 / 73.07 / 40.07 for dynamic gap.

These results provide a direct performance comparison between offline training and the dynamic strategy: dynamic gap achieves higher mean performance on two benchmarks and slightly lower performance on OptMATH. We treat this as a comparison of the final outcomes of the training pipelines and distinguish among the final corpus content, the sample-selection process, and the training order.

Regarding the motivation for comparison with a fixed curriculum, one starting point of this work is the following question: when high-quality, externally available fixed-curriculum data are limited, can we construct verifiable environments by identifying and characterizing model deficiencies, enabling the model to continually generate training examples and improve its own capabilities? External involvement is primarily used to establish the environment and verification rules. Once the environment is established, the model can generate candidate problems and solution trajectories within it, obtain supervision through verification and filtering, and perform recursive training. This reduces dependence on a continuing supply of external annotations or teacher-provided answers.

Thus, fixed-curriculum SFT with the same number of verified examples is a valuable comparison, but it primarily addresses "how to organize learning once the training data are already available." Our work also considers an earlier question: when ready-made curriculum data are scarce, how can deficiency-driven environments continually provide supervision suitable for learning? This mechanism, which starts from limited external environment design and supports subsequent self-generation and learning, is a key reason we consider the approach practically relevant in data-scarce settings.

AuthorsOpenReview ↗

Response to Reviewer 76or (2/2): External knowledge and self-improvement

Response to the concern about external knowledge and teacher supervision

Thank you for raising the boundary between external knowledge and self-improvement. We distinguish two stages: constructing an environment for discovering and characterizing deficiencies, and improving capabilities after the environment has been established.

During environment construction, large models and human input provide task structures, assistance information, and verification rules. This external knowledge turns the target capabilities into an environment in which tasks can be generated, failures observed, and outputs checked. We do not claim that the small model autonomously constructs the environment itself.

Once the environment is established, problem generation, solution-trajectory generation, and recursive training can be carried out continuously within it by the small model together with fixed generation and verification procedures. This requires neither new answers from an external large model in each round nor human specification of training content in each round. The small model produces candidate problems and trajectories, the environment provides verification feedback, the current model's performance influences subsequent training selection, and the updated model proceeds to the next round. External involvement thus supplies the initial environment and feedback conditions; subsequent improvement does not depend on a continuing supply of external teacher responses.

This also clarifies the scope of "self-improvement" in our work: the externally constructed environment makes capability deficiencies discoverable and measurable, and the learner improves these capabilities through its own generation, verification feedback, and parameter updates. Deficiency measurement relies on externally defined criteria, but the measurements vary with the current learner; capability improvements arise from the learner's recursive learning within this environment. Providing environmental structure does not mean that an external teacher performs the learner's subsequent learning on its behalf.

Reviewer_MuUJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper turns recurring optimization-modeling defects into executable environments and allocates training data using the assisted–unassisted success gap. A strong external model and human review initialize the environments. A fixed initial model generates guided, verified trajectories, and guidance is removed from the learner's training inputs. The updated learner supplies capability assessments and controlled problem rewriting. Unaided benchmark performance improves; the incremental macro gain over uniform allocation is 1.64 percentage points.

Critical strengths
  1. S1: The initialization study matches additional sampling budgets and compares targeted guidance with resampling, generic prompts, and mismatched guidance.
  2. S2: Verification checks reference outcomes and defect-specific structure rather than execution alone. The TSP example clearly motivates this distinction.
Critical weaknesses
  1. W1: The independent value of recursive allocation is not isolated. Uniform allocation is the main control; fixed R0 quotas, error-rate curricula, assisted-success curricula, and one-shot offline SFT are missing. Table 4 also lets each branch change its own quotas, mixing generator and curriculum effects (§§4.4 and 4.6).
  2. W2: Guidance contains substantial external algorithmic knowledge, including an explicit MTZ construction and control-flow scaffold. The results more directly support guided distillation and feedback-driven curricula than autonomous improvement of diagnosis, trajectory supervision, or environment construction (Appendix B.4; §5).
  3. W3: Generation, reference solving, and structural checking share a specification, allowing common semantic mistakes to pass consistently. The interpretation of free-threshold squared SOCP variants particularly warrants independent review. False acceptance/rejection rates and acceptance of alternative equivalent formulations are not reported (Appendices B.3 and C.5).
Questions for authors
  1. Q1 → W1: Can fixed R0, error-rate, assisted-success, and offline SFT baselines be tested under both equal accepted-data and equal total-compute protocols, with quotas held fixed in an additional generator comparison?
  2. Q2 → W3: Can independent semantic/feasibility audits and false acceptance/rejection rates be reported, specifically covering SOCP ambiguity and equivalent formulations?
2 comments
AuthorsOpenReview ↗

Response to Reviewer MuUJ (1/2): Allocation controls

Overview of the additional experiments

Thank you for your specific and constructive comments on our work. To address these concerns, we added three controls with matched training budgets: allocation based on the frozen initial gap, dynamic failure-rate allocation, and offline SFT. All three additional experiments use 15 rounds, 128 training records per round, an effective batch size of 8, and 240 optimizer updates, with evaluation fixed at round 15 (R15). This matches the number of training examples and the update budget of the main experiment.

The central design of our method is to translate the learner's current capability deficiencies into subsequent training decisions. The assisted condition provides a way to assess how amenable failures are to intervention; round-by-round evaluation feeds the updated model state back into training allocation; and execution-based verification checks the outputs. The resulting recursive loop is "model state → deficiency measurement → training selection → model update → remeasurement." The additional controls examine different aspects of this design, rather than treating all components as a single factor.

The mean pass@1 scores (%) on NL4Opt, MAMO, and OptMATH are as follows:

ConditionNL4Opt245MAMO203OptMATH193
Frozen initial-gap allocation93.0670.6141.45
Dynamic failure-rate allocation93.0671.2638.51
Offline SFT93.3372.5840.24
Dynamic gap (ours)93.4773.0740.07

Dynamic gap achieves the highest mean accuracy among the four conditions on NL4Opt and MAMO. Compared with the frozen initial-gap condition, it improves performance by 0.41 and 2.46 percentage points, respectively. Compared with dynamic failure-rate allocation, it improves performance by 0.41, 1.81, and 1.56 percentage points across the three benchmarks, respectively.

Thank you for suggesting that dynamic allocation, external assistance, and verification effectiveness should be examined separately. We address these issues through three additional controls and independent audits.

Response to the concern about allocation controls and computational budgets

(a) Frozen initial-gap control

The frozen condition does not use uniform random allocation. It computes the gap from the initial model's performance difference between the unaided and assisted conditions, with a 20% exploration allocation. It then continues to use this initial measurement without adjusting the target weights based on the updated model. The model is still trained in every round, and the candidate-pool capacity constraints remain in place. Thus, it is the deficiency signal that is frozen, not the model parameters; the actual quotas may still change with the remaining sample capacity.

This control retains gap information and focuses on the distinction between using initial deficiency information and continually updating that information. Dynamic gap outperforms the frozen condition by 0.41 and 2.46 percentage points on NL4Opt and MAMO, respectively. This allows us to examine the performance of dynamic feedback across tasks while retaining the initial deficiency information, rather than interpreting the frozen condition as a baseline that does not use our signal at all.

(b) Dynamic failure-rate control

The failure-rate condition allocates training examples using the current model's 1 - p_plain, while retaining dynamic evaluation, normalization, the exploration coefficient, and capacity projection. It asks whether the assisted condition provides useful additional allocation information when the method already dynamically prioritizes tasks on which the model fails.

The motivation for the gap is to distinguish failures that differ in their responsiveness to assistance. The failure rate measures "how often the model currently fails," whereas the assistance gap describes "how performance changes when structural support is provided." In the results above, dynamic gap exceeds the failure-rate condition by 0.41, 1.81, and 1.56 percentage points in mean performance across the three benchmarks, respectively.

AuthorsOpenReview ↗

Response to Reviewer MuUJ (2/2): Budgets, external structure, and verification

(c) Offline SFT and computational budgets

We added offline SFT as a reference that does not update sample-selection decisions during training. Its scores are 93.33 / 72.58 / 40.24, compared with 93.47 / 73.07 / 40.07 for dynamic gap.

The three additional experiments use the same dual-A800 configuration, effective batch size, and update budget. The frozen, failure-rate, and offline conditions process approximately 5.264 million, 5.298 million, and 4.886 million training input tokens, respectively. The failure-rate condition also performs assisted measurements to match the feedback-query design of the original dynamic control, but uses only the failure rate for allocation. We report training and feedback costs separately to avoid equating an identical sample count with identical end-to-end computational cost.

These additional experiments primarily examine allocation and training within a fixed pool of candidate trajectories, helping reduce the influence of trajectory-generation randomness on the comparisons. Allocation based solely on the assisted success rate is not included in this set of controls. Earlier generator-refresh experiments involved multiple changes to the pipeline; their results should therefore be interpreted in terms of the corresponding overall configurations, rather than as evidence of an isolated generator effect.

Response to the concern about external scaffolding and recursive learning

Scaffolds such as MTZ provide explicit algorithmic structure and form part of the assisted condition. Their role is to reveal how the model performs when given specific support, whereas round-by-round feedback from the current learner determines subsequent training selection. External structure and feedback-driven learning therefore serve different purposes. The recursive nature of our method comes from the updated model's influence on the next round's training decisions, not from the model independently inventing all algorithmic knowledge.

Response to the concern about verifier validity and semantic checks

We independently recomputed the numerical solutions for 48 SOCP DEV instances using a standard-cone formulation, reproduced all reference objective values within a tolerance of 1e-4, and checked feasibility. This provides independent support for the reference values under the intended standard-cone interpretation. Four constructed programs that only printed known objective values all passed objective-value matching but were rejected by the joint verifier. This demonstrates that, in these cases, the joint checks can exclude programs that "only output the correct numerical value."

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.