acceptodds
Under review as a conference paper at ICLR 2027

A Model Can Help Itself: Reward-Free Self-Training for LLM Reasoning

Abstract

Can a language model improve reasoning by learning from its own imperfect responses, without rewards or teacher-provided solutions? We present Self-evolving Post-Training (SePT), a simple method that alternates temperature-controlled self-generation with next-token likelihood training. Each round uses the updated model to generate new training responses, with one response per prompt by default and no correctness filtering. Across six mathematical benchmarks, SePT improves a temperature-selected no-training baseline by 11.4 and 6.7 AVG points on Qwen2.5-Math-7B and Qwen2.5-7B, respectively, where AVG averages Pass@1, Pass@8 and Pass@32 across benchmarks. We analyze how sampling temperature shapes the learning signal and investigate the value of the resulting responses. Responses from a SePT-trained model improve a student initialized from the original weights, while their reasoning prefixes help an unchanged model complete solutions, even when matched in length to prefixes from a colder initial model. Comparing next-token predictions at identical contexts also reveals changes in token rankings that decoding-temperature adjustment cannot reproduce. Further evaluations across nine starting models, general reasoning and code generation examine broader applicability. Together, these results show that reward-free self-training can improve both a model's predictions and the supervision it provides.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, reasoning, self-training

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_9wufOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Self-evolving Post-Training (SePT), where a model generates responses and then trains on them using a next-token likelihood objective. By default, it uses one response per prompt, with no rewards, correctness filtering, or external teacher solutions during training. The sampling temperature gradually increases toward one, and the generator is updated after each batch of training updates. The authors evaluate nine starting models on six math benchmarks, along with additional experiments on general reasoning and code generation. The largest gains are on Qwen2.5-Math-7B and Qwen2.5-7B, which improve by 11.4 and 6.7 AVG points over baselines selected by sampling temperature. The generator-transfer and prefix experiments also suggest that SePT produces more useful responses: these responses help both a student reset to the original weights and an unchanged model completing partial solutions. Results on the other models and coding tasks are more mixed.

Critical strengths
  1. SePT trains on all sampled responses without a verifier or correctness filtering. On Qwen2.5-Math-7B, it improves Pass@1 by 16.8 points and Pass@32 by 6.9 points over the temperature-selected baseline (Table 1). The gains also hold across the reported data-order runs and with the lower-overlap OTM training set (Tables 12 and 27). It is interesting that self-generated data can improve a model even when the responses are not checked for correctness.

  2. The authors compare SePT responses against fresh responses from the initial model, including a setting where the student is reset to its original weights before training. When both generators use temperature 0.3, the student trained on SePT responses gets 67.1% MATH-500 Pass@1, compared with 50.2% for the initial-generator control (Table 4). Length-matched SePT prefixes also help an unchanged model complete solutions, reaching 68.46% accuracy compared with 57.83% when solving the same eligible questions from scratch (Table 5). These controls give fairly direct evidence that the generated text becomes more useful, both as training data and as a starting point for solving problems.

Critical weaknesses
  1. The authors mention in the appendix that the baseline temperature and SePT checkpoint are selected using the same benchmark AVG that is later reported. The bootstrap intervals hold those choices fixed, so they do not capture uncertainty from selection. It is also unclear how many math checkpoints were evaluated and how often. This makes it harder to judge the size of the reported gains, especially the smaller differences on other models. A separate validation set or a prespecified training endpoint would make the comparison more convincing. The fixed-endpoint generator controls provide supporting evidence, but they do not address this concern for Table 1.

  2. The main EM-FT comparison changes both the objective and the sampling schedule: EM-FT samples at temperature one throughout, while SePT gradually increases the temperature toward one. Table 32 helps by comparing the objectives on shared contexts, but the Pass@1 difference is only 0.18 points, with a 95% interval of [−1.78, 2.13]. Likelihood fitting does improve Pass@8 by 4.20 points, which supports a benefit when sampling multiple answers. Still, the entropy arm clips gradients on 82.5% of updates and does not have separate loss-coefficient or learning-rate tuning. I am curious how much of the overall gain comes from the likelihood objective, the sampling schedule, or the interaction between them. Given the close connection to entropy minimization, this is important for understanding the paper’s added contribution.

Questions for authors
  1. I am curious how many checkpoints and training configurations were considered for each model in Table 1, and how often were checkpoints evaluated?
  2. Was a separate validation set used? If endpoint results are available, could you report the comparison using the final checkpoint or another prespecified checkpoint? This would help show how much the gains depend on selection using benchmark AVG.
  3. Did you run EM-FT with SePT’s sampling schedule, or tune the entropy coefficient or learning rate separately?
  4. Table 32 shows almost identical Pass@1 on shared contexts, but a clearer Pass@8 advantage for likelihood fitting. I am curious if any existing results help separate the effect of the objective from the sampling schedule and optimization settings?
3 comments

Thank you for your careful review and suggestions on model selection and objective comparisons.

  1. How many checkpoints/configurations were considered, and how often were they evaluated?

    The original nine-model comparison uses a common SePT recipe: AdamW with learning rate 10−710^{-7}, one response per question and two updates per generated batch. Sampling starts at 0.6 for Qwen2.5-Math and DeepSeek-Math-Instruct, and 0.9 for the others, rising toward one over two epochs (Section 3.1).

    For submitted Table 1, we use a 900-step checkpoint-selection window, saving and evaluating every 50 steps. We report the checkpoint with the highest benchmark AVG within that window.

  2. Was validation separate, and do gains hold at a fixed training endpoint?

    Original protocol. Submitted Table 1 uses the reported benchmarks for selection; its bootstrap intervals condition on those choices. This follows common benchmark-guided development practice in reasoning post-training (e.g., DAPO [1]). We now add both independent validation and a training duration fixed before evaluation.

    New validation and endpoint results. We reserve 256 MATH training-split questions, stratified across seven subjects and difficulty levels 1–5 without model scores. Exact/near-match checks separate them from the six test benchmarks and remove matching validation/test questions from DSR training. Validation alone selects temperature, hyperparameters and the checkpoint; checkpoints are saved and evaluated every 25 rounds. These comparisons train for 300 rounds, fixed beforehand.

    SettingValidation-selected test AVGFinal (300 rounds) test AVG
    Baseline, no training42.1442.14
    SePT, original learning rate/schedule51.43 (+9.29)52.46 (+10.32)
    SePT, validation-tuned learning rate53.77 (+11.63)54.41 (+12.27)

    AVG averages Pass@1/8/32 across the paper's six benchmarks, with 32 responses per question. Validation selects Baseline temperature 0.4; trained models use τeval=1\tau_{\mathrm{eval}}=1. The original learning rate is 10−710^{-7}; tuning chooses 3×10−73\times10^{-7} with the same schedule, averaged over two training runs. Substantial gains hold without checkpoint selection and with the original learning rate and sampling schedule (Section 3.7; Appendix L.1).

    Beyond the principal Qwens. We also added text-only Gemma4 to the main-table comparison:

    ModelMethodPass@1Pass@8Pass@32AVG
    E4B-ITBaseline21.4361.4575.4652.78
    E4B-ITSePT48.7368.8774.9564.18 ± 0.19
    E2B-ITBaseline18.6353.3770.1547.38
    E2B-ITSePT31.2562.8270.6554.91 ± 0.40

    Gains are +11.40/+7.53 AVG points, with improvement in all three training runs (mean ± SD). Both methods use 32 responses and a 6,144-token cap; original-weight Baseline temperature is 1.5, SePT one (Table 1; Appendix L.3).

    [1] Qiying Yu et al. DAPO: An Open-Source LLM Reinforcement Learning System at Scale. 2025

We continue with your questions on the entropy objective and sampling schedule.

  1. Did EM-FT use SePT's schedule and separate tuning?

    We added this control as SePT-EM: SePT with entropy minimization replacing the likelihood objective. Both objectives use Qwen2.5-Math-7B and the same filtered DSR questions, with 38,400 responses and 600 updates. Sampling temperature rises from 0.6 toward one over 628 intervals, reaching about 0.79 at round 300; training temperature is one. Each model generates its own responses: SePT-EM minimizes next-token entropy on those prefixes; SePT fits the sampled tokens.

    Tuning and clipping. Four SePT-EM settings combine learning rates {10−7,3×10−8}\{10^{-7},3\times10^{-8}\} with entropy coefficients {1,0.1}\{1,0.1\}. Held-out validation chooses 10−710^{-7} and coefficient 1. Coefficient 0.1 reduces clipping to 10–26% of updates but gives lower validation scores, directly testing the optimization concern.

    Results. These are validation-selected six-benchmark test scores averaged over two training runs, with 32 responses per question at τeval=1\tau_{\mathrm{eval}}=1:

    MethodPass@1Pass@8Pass@32AVG
    SePT37.9756.9566.4053.77
    SePT-EM33.8751.7260.9048.83

    SePT improves all three Pass metrics, at similar training cost: 36.97 versus 36.96 GPU-hours (Appendix L.2).

  2. What separates the objective, schedule and optimization effects?

    Objective at matched settings. In a paired comparison with the same learning rate 10−710^{-7} and sampling schedule, SePT exceeds SePT-EM by 2.14 test AVG points after validation selection. SePT-EM has slightly higher Pass@1 in this same-learning-rate comparison; Q3 gives the separately tuned comparison.

    Identical contexts. Submitted Table 32 shows +4.20 Pass@8 points [2.00, 6.60] and similar Pass@1, with a difference of +0.18 [−1.78, 2.13]. The new online comparison above complements that experiment by allowing each model to generate its own training responses. Brackets denote 95% question-bootstrap intervals.

    Schedule and learning rate. Holding likelihood training fixed, submitted Table 3 changes constant sampling temperature one to SePT's schedule and improves test AVG from 44.6 to 55.0. In the new fixed-schedule learning-rate control, changing 10−710^{-7} to 3×10−73\times10^{-7} improves validation AVG at round 100 from 88.10 to 91.35 (Appendix L.2).

    Together these controls show contributions from both the sampling schedule and optimization, while the matched, tuned comparison establishes a likelihood-training advantage. The generator-transfer and prefix experiments identify what is learned: improved responses that help a reset student and an unchanged completer, beyond a decoding-temperature adjustment.

Dear Reviewer 9wuf,

We would like to gently follow up on our rebuttal, which includes validation and fixed-endpoint results, the controlled entropy-minimization comparison, and strong Gemma4 gains beyond Qwen. When you have a moment, could you please take a look and share your thoughts? Thank you again for your time and consideration.

Reviewer_kVUdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SePT alternates generating unfiltered responses from the current model with next-token likelihood training, without correctness rewards or reference solutions. In the main mathematical recipe, it uses one response per prompt, gradually raises the sampling temperature to one, and regularly refreshes the generator. Across six mathematical benchmarks, Qwen2.5-Math-7B and Qwen2.5-7B improve AVG—the mean of Pass@1, Pass@8, and Pass@32—over temperature-selected no-training baselines by 11.4 and 6.7 points, respectively; results on seven other starting models are less conclusive. Fixed-generator controls, reset-student transfer, prefix interventions, and token-ranking comparisons investigate the value of learned supervision beyond decoding changes. Further experiments examine data overlap, truncation, and broader applicability, with coding gains uncertain relative to a tuned baseline and substantial degradation under longer hot-source training.

Critical strengths
  1. A simple reward-free recipe produces substantial gains in the principal Qwen models. SePT improves AVG over temperature-tuned baselines by 11.4 points on Qwen2.5-Math-7B and 6.7 points on Qwen2.5-7B. Repeated runs and lower-overlap training data support these gains, while decoder and truncation controls strengthen the evidence that unfiltered self-generated responses can provide useful training signals.
  2. The source/student controls reveal value in the learned supervision beyond the final model’s weights. A frozen SePT generator teaches a student reset to its original weights, outperforming an initial-generator control by 24.25 MATH-500 Pass@1 points; the advantage remains 16.95 points at a shared sampling temperature. Length-matched prefixes also improve an unchanged completer, supporting the usefulness of generated content without claiming a complete causal explanation.
Critical weaknesses
  1. Direct comparisons with alternative self-training methods are limited. The evaluation focuses on EM-FT and GRPO, without quantitative comparisons to label-free reward-based methods such as TTRL. Although these methods use different supervision signals, they would help establish the tradeoff between SePT’s reward-free simplicity and alternative ways of learning from self-generated data.
  2. Effectiveness and stability vary substantially across models and tasks. Clear AVG gains are concentrated in two Qwen models, while the other seven confidence intervals include zero and Phi’s Pass@1 declines significantly. Coding gains over the tuned baseline remain uncertain, and longer hot-source training causes substantial degradation. The paper therefore provides limited guidance on which models benefit reliably and when training should stop.
Questions for authors
  1. Could you include additional self-training baselines, particularly label-free methods using consensus rewards? How does SePT compare in performance, compute cost, and stability, and when is reward-free training preferable?
  2. What properties of the starting models explain the variation in SePT’s gains? Why are clear AVG improvements concentrated in the two principal Qwen models?
  3. To what extent do the differences between the math and coding results reflect the tasks themselves versus their different sampling-temperature schedules? Can the existing experiments distinguish these effects?
2 comments

Thank you for your thoughtful review and questions on self-training.

  1. Consensus-reward comparison: performance, cost, stability and when to prefer SePT.

    Comparison. We added a TTRL-style consensus GRPO baseline [1] on Qwen2.5-Math-7B, using the same DSR questions, 38,400 responses and 600 updates. Each round uses 128 questions × 1 response for SePT, versus 8 × 16 for consensus. Consensus samples at temperature one. Both select checkpoints on held-out validation. We show the paired run with the higher SePT test AVG among the completed repeats. Scores use the paper's six math benchmarks, 32 responses per question and τeval=1\tau_{\mathrm{eval}}=1; AVG averages Pass@1/8/32 across benchmarks.

    MethodPass@1Pass@8Pass@32AVGTraining GPU-hours
    SePT39.1957.7867.0954.6937.30
    Consensus GRPO37.9859.5666.1754.5740.03

    SePT is competitive with 6.83% less training GPU time in this pair.

    When reward-free training is preferable. Consensus needs an answer extractor and an equivalence rule to count votes, even without ground-truth labels. These are convenient for short mathematical answers but harder for open-ended outputs. For example, two correct programs may use different algorithms and share no code. Extracting their code blocks does not establish whether they compute the same function; grouping by behavior requires execution inputs and a rule for comparing outputs. SePT trains directly on either response, using one response per question without grouping answers or assigning rewards. It is therefore particularly useful when reliable answer comparison is unavailable or costly.

    [1] Yuxin Zuo et al. TTRL: Test-Time Reinforcement Learning. 2025.

  2. Which starting-model properties matter, and are gains confined to Qwen?

    Properties and evidence. We hypothesize that gains depend on how sampling changes the starting model's response quality and whether training retains its solution diversity. The new analysis in Appendix L.3 finds that the initial sampling settings raise untrained Pass@1 by 6.72 points for Qwen2.5-Math and 3.30 for Qwen2.5, versus 0.35 for DeepSeek-Math. This suggests greater scope to improve their initial training responses through sampling. The accuracy–coverage balance also matters: both principal Qwens improve Pass@1 and Pass@32, whereas Qwen3 gains 5.00 Pass@1 points but loses 2.58 Pass@32 points.

    New evidence beyond Qwen. Text-only Gemma4 also shows strong gains on the same six benchmarks:

    ModelMethodPass@1Pass@8Pass@32AVG
    E4B-ITBaseline21.4361.4575.4652.78
    E4B-ITSePT48.7368.8774.9564.18 ± 0.19
    E2B-ITBaseline18.6353.3770.1547.38
    E2B-ITSePT31.2562.8270.6554.91 ± 0.40

    Gains are +11.40/+7.53 AVG points, with improvement in all three training runs (mean ± SD). Baseline uses original weights at the selected temperature 1.5; SePT uses τeval=1\tau_{\mathrm{eval}}=1, with 32 responses and a 6,144-token cap for both (Table 1; Appendix L.3). Their sampling starts above one. A new E4B control also gains 11.27 AVG points from updating rather than freezing the generator, averaged over two runs, extending the paper's mechanism evidence to this family.

  3. Can existing experiments distinguish task effects from temperature schedules?

    They isolate a schedule effect within coding. Holding Qwen3-4B-Instruct-2507, rStar-Coder questions, learning rate and 64-round duration fixed, starting τs\tau_s at 0.9 rather than 1.2 improves CodeScore by 22.93 points [19.33, 26.62]. Both schedules move toward one. CodeScore averages Pass@1/5 over LiveCodeBench and MBPP+; brackets are 95% bootstrap intervals.

    A shorter 16-round repeat of the above-one schedule gives +1.19 [−0.32, 2.72] over its tuned Baseline. Together, these within-task comparisons identify sampling schedule and duration as concrete contributors to the coding results. They support adapting the schedule to the starting model and using validation to choose training duration (Appendix K.1).

Dear Reviewer kVUd,

We would like to gently follow up on our rebuttal, which includes the consensus-reward comparison, sampling-schedule analysis, and strong Gemma4 gains beyond Qwen. When you have a moment, could you please take a look and share your thoughts? Thank you again for your time and consideration.

Reviewer_9LEqOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Self-evolving Post-Training (SePT), a reward-free self-training method that repeatedly samples responses from the current language model, trains on the unfiltered responses with next-token likelihood, and refreshes the generator after each update. The paper analyzes sampling and training temperatures, compares evolving and frozen generators, studies transfer of generated responses and reasoning prefixes, and evaluates nine starting models across mathematical reasoning, general reasoning, and code-generation tasks. The strongest gains are reported for Qwen2.5-Math-7B and Qwen2.5-7B.

Critical strengths
  1. The paper asks a clear and meaningful question and proposes a simple training procedure rather than a complicated pipeline. The temperature-dynamics analysis and the discussion of shared-parameter transfer provide useful conceptual insight.\n\n2. The generator-control, frozen-generator transfer, prefix intervention, overlap audit, and repeated data-order experiments make the paper substantially more informative than a single benchmark comparison.
Critical weaknesses
  1. The evaluation protocol appears to select the baseline temperature and SePT checkpoint using the reported evaluation benchmarks. If the final test questions are used for either selection, the main comparison is affected by test-set selection bias and requires a new held-out evaluation.\n\n2. The generality claim is weaker than the headline results suggest: statistically clear gains are concentrated in two Qwen models, while most other models have confidence intervals overlapping zero and one model has a negative point estimate. The method also requires repeated generation and training, but equal-token or equal-FLOP comparisons with temperature scaling, entropy minimization, and on-policy distillation are not sufficiently established.
Questions for authors
  1. Were the final evaluation questions used to select the baseline temperature or SePT checkpoint? If so, can the authors provide results using an independent validation set for all selection decisions?\n\n2. What are the results after removing all possible overlaps between the training datasets and the evaluation benchmarks?\n\n3. How does SePT compare with entropy minimization and on-policy distillation under the same generated-token and optimization budgets?\n\n4. How does the training objective change when sequence-sum and token-averaged likelihood are used under otherwise identical settings?
2 comments

Thank you for your careful review and suggestions on evaluation.

  1. Were test questions used for selection, and do gains survive independent validation?

    For submitted Table 1, we selected the Baseline temperature and SePT checkpoint on the reported benchmarks. This follows common benchmark-guided development practice in reasoning post-training (e.g., DAPO [1]). To test the gains with independent selection, we conducted a new experiment using a separate validation set.

    For Qwen2.5-Math-7B, we reserve 256 MATH training-split questions, stratified across seven subjects and difficulty levels 1–5 without using model scores. We exclude exact/near matches to the six test benchmarks and remove DSR questions matching validation or test questions. Validation questions and answers are excluded from training. Validation selects the Baseline temperature, hyperparameters and highest-scoring checkpoint; checkpoints are saved and evaluated every 25 rounds (Appendix L.1).

    SettingSix-benchmark test AVGGain
    Baseline, no training42.14—
    SePT, original learning rate/schedule51.43+9.29
    SePT, validation-tuned learning rate53.77+11.63

    AVG averages Pass@1/8/32 across the paper's six benchmarks, with 32 responses per question. Baseline uses τeval=0.4\tau_{\mathrm{eval}}=0.4; trained models use one. Tuning changes the learning rate from 10−710^{-7} to 3×10−73\times10^{-7}, keeping the schedule. Tuned SePT scores average two training runs (Section 3.7).

    [1] Qiying Yu et al. DAPO: An Open-Source LLM Reinforcement Learning System at Scale. 2025, §4.

  2. Results after removing all candidate training–test overlaps.

    This experiment is already in submitted Appendix C.1/Table 28: removing all 1,310 flagged DSR questions, including the 163 subsequently confirmed matches, gives 38.11 → 55.23 AVG (+17.12). Both the original weights and SePT are evaluated at τeval=1\tau_{\mathrm{eval}}=1.

  3. Entropy minimization and on-policy distillation under matched budgets.

    Objective and cost. All methods use Qwen2.5-Math-7B, the same filtered DSR questions, 38,400 generated responses and 600 updates. SePT-EM replaces SePT's likelihood objective with entropy minimization, keeping its online generation and sampling schedule; its learning rate/coefficient are tuned separately. OPD uses a frozen Qwen2.5-7B-Instruct teacher with a top-128 approximation. Checkpoints are validation-selected.

    MethodTest AVGTraining GPU-hoursTokens (M)
    SePT53.7736.9735.42
    SePT-EM48.8336.9637.44
    OPD51.0067.9266.94

    SePT and SePT-EM average two runs; OPD uses one. SePT exceeds SePT-EM on all three Pass metrics at similar cost, and achieves higher AVG than OPD with about 46% less training GPU time. Costs include full-run generation, optimization and teacher computation (Appendix L.2).

  4. How do sequence-sum and token-averaged likelihood differ?

    For token loss ℓit\ell_{it}, response length LiL_i and batch size BB,

    Lseq=1B∑i,tℓit,Ltok=1∑iLi∑i,tℓit.L_{\mathrm{seq}}=\frac{1}{B}\sum_{i,t}\ell_{it},\qquad L_{\mathrm{tok}}=\frac{1}{\sum_iL_i}\sum_{i,t}\ell_{it}.

    On an identical batch, Lseq=LˉLtokL_{\mathrm{seq}}=\bar L L_{\mathrm{tok}}: raw gradients differ by mean response length Lˉ\bar L. Across batches, this random length normalization changes the expected weighting. Appendix F.2 derives the resulting target distribution: the expected token-averaged gradient can be nonzero even with identical generator/student weights and τs=τt\tau_s=\tau_t. Thus token averaging changes both scale and the expected objective.

Additional evidence addressing your generality concern.

Your second weakness notes that clear gains were concentrated in two Qwen models. We therefore evaluated text-only Gemma4 and found strong gains in both models:

ModelMethodPass@1Pass@8Pass@32AVG
E4B-ITBaseline21.4361.4575.4652.78
E4B-ITSePT48.7368.8774.9564.18 ± 0.19
E2B-ITBaseline18.6353.3770.1547.38
E2B-ITSePT31.2562.8270.6554.91 ± 0.40

Gains are +11.40/+7.53 AVG points over temperature-selected Baselines, with improvement in all three training runs (mean ± SD). Both methods use 32 responses and a 6,144-token cap; Baseline temperature is 1.5, SePT one (Table 1; Appendix L.3).

Dear Reviewer 9LEq,

We would like to gently follow up on our rebuttal, which includes independent validation results, additional baselines, and strong Gemma4 gains beyond Qwen. When you have a moment, could you please take a look and share your thoughts? Thank you again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.