acceptodds
Under review as a conference paper at ICLR 2027

Learning from Successes and Failures: Efficient Self-Evolution through Asymmetric Supervision

Abstract

Reasoning models can continue to improve after instruction tuning, yet existing approaches often either use reinforcement learning for self-evolution or rely on pre-existing per-question data for supervised refinement. We ask whether an instruction-tuned model can improve by directly learning from the structure of its own generated experience. Our key insight is asymmetric supervision: verified successes provide constructive targets for SFT, while competing incorrect answers provide comparative signals without treating failed reasoning trajectories as negative demonstrations. Starting only from a high-level domain prior and difficulty specifications, our framework constructs capability-matched curricula from model-generated problems and converts verified experience into success amplification and answer-level failure discrimination, using parameter-efficient LoRA updates on a compact training set. On Qwen3-4B and Qwen3-8B, verified-success SFT alone improves average accuracy across five challenging mathematical benchmarks by 2.27 and 2.28 percentage points, demonstrating that self-improvement without reinforcement learning is possible. Failure-aware supervision provides further gains, yielding total improvements of 3.12 and 2.97 percentage points, respectively, with aggregate gains across a broader nine-benchmark mathematical reasoning suite. Repeated-sampling analyses further indicate improved access to successful solutions.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Autonomous Self-Evolution, Asymmetric Supervision, Self-Verification

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_jWCkOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether a reasoning model can improve using only problems and solutions generated by itself, without reinforcement learning or an existing question-level training set. The method first trains a Challenger to generate questions near the current ability level of the model. The original model then solves each question several times and groups the answers into clusters. Verified successful trajectories are used for supervised fine-tuning, while repeated wrong answers are used in an answer-level pairwise loss. Experiments on Qwen3-4B and Qwen3-8B show improvements of about 2–3 points on several math benchmarks.

Critical strengths
  • The paper gives a clear reason for treating successful and failed samples differently. Learning complete successful trajectories while only comparing the final answers of failed samples avoids penalizing useful intermediate reasoning.

  • The method does not require reinforcement learning or a human-written question set. The Challenger and Solver are also separated, which makes the training process easier to understand.

  • The paper includes useful ablations on question selection, verification, the number of positive samples, and the use of failed trajectories. The Pass@k and problem-state results also provide more information than single-sample accuracy alone.

Critical weaknesses
  • Limited novelty. The method mainly combines capability-matched question generation, SFT on successful trajectories, and answer-level preference learning. The failure-aware loss is also close to existing reference-free preference objectives. The overall contribution is useful, but it is mostly a combination of known techniques.

  • Limited experimental scope and modest gains. The experiments only use Qwen3-4B and Qwen3-8B, and the training data are limited to mathematics. The full method improves the average scores by only about 2–3 points, while performance drops on some individual benchmarks. It is unclear whether the method works well across other model families and tasks.

  • Too few baselines. The main experiments only compare Initial, Success-SFT, and Full. The paper lacks direct comparisons with recent related methods, such as [1] Self-Questioning Language Models, [2] Guided Self-Evolving LLMs with Minimal Human Supervision, [3] R-Diverse: Mitigating Diversity Illusion in Self-Play LLM Training, [4] Self-Distillation Zero: Self-Revision Turns Binary Rewards into Dense Supervision. Although their settings are not exactly the same, comparisons under the same backbone, question set, and compute budget would make the results more convincing.

  • Missing experimental details. The paper does not clearly report how many samples remain after each filtering stage, the final size of the training set, the accuracy of self-verification, or the total cost of data generation and training. It also does not give the number of evaluation runs, variance, or confidence intervals. These details are needed to judge reproducibility, statistical reliability, and the claim of efficiency.

Questions for authors
  1. How many questions and trajectories remain after each filtering and verification stage? What is the final training-set size for each model2
  2. How many evaluation runs were used for each result? Could the authors report standard deviations, confidence intervals, or significance tests for the 2–3 point improvements?
  3. What are the total costs of question generation, repeated sampling, verification, and training? How do these costs compare with recent self-evolution or RL-based methods?
  4. Can the authors provide direct comparisons with recent methods such as [1] Self-Questioning Language Models, [3] DARC, [5] Self-Distillation Zero, or [6] Self-Verified Distillation under a matched backbone and training budget? Such results could materially change my view of the paper’s novelty and effectiveness.
Reviewer_DbTdOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper suggest a training schema which trains the model without reinforcement learning and training dataset. Main novelty lies in how to substitute RL in zero-data regime, since existing works either require RL or supervision dataset. They give asymmetric supervision over success trajectory and failure trajectory during SFT.

Critical strengths

The idea is simple and interesting.

Critical weaknesses
  1. There is no baseline model that supports RL without dataset vs SFT with dataset (self-training) vs ours. Models such as Absolute Zoer, R-Zero, TTRL / STaR, ScPO, RFT should be baseline models to prove that SFT without dataset is helpful.
  2. Also, they do not report standard deviation from multiple seed evaluation, and this is of great concern because the performance gap is very small. Also, the evaluation dataset size is very small, therefore multiple runs is a must.
Questions for authors

The training configurations seems to update very little, considering that it is a LoRA. The learning rate must be much higher to show some training effect. If the authors have done multiple samplings, please report the standard deviation. Show the results of baselines such as RFT/STaR, SVD, R-Zero, DPO/SimPO.

Reviewer_EamQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The setting is self-improvement without RL. The model writes its own problems, keeps the ones whose answers it can reproduce at a middling rate rather than always or never, and freezes 2000 of them into a curriculum. It then solves each problem several times, groups the attempts by final answer, and runs its own staged checks over the majority group. Two terms are trained on a LoRA adapter: plain SFT on one checked solution per problem, and a pairwise term that pushes the checked answer above the competing wrong answers, with the same reasoning prefix on both sides and weights set by how often each wrong answer came up. Failed reasoning traces are never used as negatives, which is what the authors mean by asymmetric. On Qwen3-4B and 8B the SFT term alone is worth about 2.3 points on the five hard math benchmarks, 3.1 and 3.0 with the pairwise term added, and roughly 2 points on the nine-benchmark average. There are ablations on the failure side of the objective and on the curriculum filters, plus a repeated-sampling analysis.

Critical strengths
  1. The paper's key design choice is to contrast answers rather than full trajectories. The ablation in Table 4 shows that this matters, with performance dropping from 61.00 to 59.19 on Hard-5.

  2. The authors are also upfront about where their method falls short. Full underperforms Success-SFT at k = 4, 8, and 10, which makes the overall results more convincing to me.

Critical weaknesses
  1. Nothing is compared against except the starting checkpoint. No RL method appears in any table, no compute is reported, and the recipe is never run on externally sourced problems - the No Challenger and Random variants in Table 5 are still self-generated. So I cannot tell whether the 3.12 comes from asymmetric supervision, from self-generation, or from LoRA SFT on 2000 self-verified solutions.

  2. Single runs on small benchmarks. One problem moves Hard-5 by 0.67, and the pairwise term is worth 0.85 and 0.69 points, i.e. about one problem. No seed count, no spread, no evaluation decoding settings. The headline gain is probably real; the paper's second contribution is not separable from noise as reported.

Questions for authors
  1. What decoding settings and how many samples per problem are behind Table 1? AIME24 at 72.33 is not reachable with one sample, and Sec. 4.1 promises settings that I could not find.
  2. Run the identical Success-SFT and Full recipe on an external problem pool of the same size - MATH train, NuminaMath - and report Hard-5. How much of the 3.12 survives? This is the experiment that would most change my score.
  3. On a labelled subset, what fraction of accepted trajectories have a wrong final answer?
  4. How many of the 2000 curriculum problems survive all three filters?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.