- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces Self-evolving Post-Training (SePT), where a model generates responses and then trains on them using a next-token likelihood objective. By default, it uses one response per prompt, with no rewards, correctness filtering, or external teacher solutions during training. The sampling temperature gradually increases toward one, and the generator is updated after each batch of training updates. The authors evaluate nine starting models on six math benchmarks, along with additional experiments on general reasoning and code generation. The largest gains are on Qwen2.5-Math-7B and Qwen2.5-7B, which improve by 11.4 and 6.7 AVG points over baselines selected by sampling temperature. The generator-transfer and prefix experiments also suggest that SePT produces more useful responses: these responses help both a student reset to the original weights and an unchanged model completing partial solutions. Results on the other models and coding tasks are more mixed.
Critical strengths
-
SePT trains on all sampled responses without a verifier or correctness filtering. On Qwen2.5-Math-7B, it improves Pass@1 by 16.8 points and Pass@32 by 6.9 points over the temperature-selected baseline (Table 1). The gains also hold across the reported data-order runs and with the lower-overlap OTM training set (Tables 12 and 27). It is interesting that self-generated data can improve a model even when the responses are not checked for correctness.
-
The authors compare SePT responses against fresh responses from the initial model, including a setting where the student is reset to its original weights before training. When both generators use temperature 0.3, the student trained on SePT responses gets 67.1% MATH-500 Pass@1, compared with 50.2% for the initial-generator control (Table 4). Length-matched SePT prefixes also help an unchanged model complete solutions, reaching 68.46% accuracy compared with 57.83% when solving the same eligible questions from scratch (Table 5). These controls give fairly direct evidence that the generated text becomes more useful, both as training data and as a starting point for solving problems.
Critical weaknesses
-
The authors mention in the appendix that the baseline temperature and SePT checkpoint are selected using the same benchmark AVG that is later reported. The bootstrap intervals hold those choices fixed, so they do not capture uncertainty from selection. It is also unclear how many math checkpoints were evaluated and how often. This makes it harder to judge the size of the reported gains, especially the smaller differences on other models. A separate validation set or a prespecified training endpoint would make the comparison more convincing. The fixed-endpoint generator controls provide supporting evidence, but they do not address this concern for Table 1.
-
The main EM-FT comparison changes both the objective and the sampling schedule: EM-FT samples at temperature one throughout, while SePT gradually increases the temperature toward one. Table 32 helps by comparing the objectives on shared contexts, but the Pass@1 difference is only 0.18 points, with a 95% interval of [−1.78, 2.13]. Likelihood fitting does improve Pass@8 by 4.20 points, which supports a benefit when sampling multiple answers. Still, the entropy arm clips gradients on 82.5% of updates and does not have separate loss-coefficient or learning-rate tuning. I am curious how much of the overall gain comes from the likelihood objective, the sampling schedule, or the interaction between them. Given the close connection to entropy minimization, this is important for understanding the paper’s added contribution.
Questions for authors
- I am curious how many checkpoints and training configurations were considered for each model in Table 1, and how often were checkpoints evaluated?
- Was a separate validation set used? If endpoint results are available, could you report the comparison using the final checkpoint or another prespecified checkpoint? This would help show how much the gains depend on selection using benchmark AVG.
- Did you run EM-FT with SePT’s sampling schedule, or tune the entropy coefficient or learning rate separately?
- Table 32 shows almost identical Pass@1 on shared contexts, but a clearer Pass@8 advantage for likelihood fitting. I am curious if any existing results help separate the effect of the objective from the sampling schedule and optimization settings?
3 comments
Part 2/2
We continue with your questions on the entropy objective and sampling schedule.
-
Did EM-FT use SePT's schedule and separate tuning?
We added this control as SePT-EM: SePT with entropy minimization replacing the likelihood objective. Both objectives use Qwen2.5-Math-7B and the same filtered DSR questions, with 38,400 responses and 600 updates. Sampling temperature rises from 0.6 toward one over 628 intervals, reaching about 0.79 at round 300; training temperature is one. Each model generates its own responses: SePT-EM minimizes next-token entropy on those prefixes; SePT fits the sampled tokens.
Tuning and clipping. Four SePT-EM settings combine learning rates with entropy coefficients . Held-out validation chooses and coefficient 1. Coefficient 0.1 reduces clipping to 10–26% of updates but gives lower validation scores, directly testing the optimization concern.
Results. These are validation-selected six-benchmark test scores averaged over two training runs, with 32 responses per question at :
Method Pass@1 Pass@8 Pass@32 AVG SePT 37.97 56.95 66.40 53.77 SePT-EM 33.87 51.72 60.90 48.83 SePT improves all three Pass metrics, at similar training cost: 36.97 versus 36.96 GPU-hours (Appendix L.2).
-
What separates the objective, schedule and optimization effects?
Objective at matched settings. In a paired comparison with the same learning rate and sampling schedule, SePT exceeds SePT-EM by 2.14 test AVG points after validation selection. SePT-EM has slightly higher Pass@1 in this same-learning-rate comparison; Q3 gives the separately tuned comparison.
Identical contexts. Submitted Table 32 shows +4.20 Pass@8 points [2.00, 6.60] and similar Pass@1, with a difference of +0.18 [−1.78, 2.13]. The new online comparison above complements that experiment by allowing each model to generate its own training responses. Brackets denote 95% question-bootstrap intervals.
Schedule and learning rate. Holding likelihood training fixed, submitted Table 3 changes constant sampling temperature one to SePT's schedule and improves test AVG from 44.6 to 55.0. In the new fixed-schedule learning-rate control, changing to improves validation AVG at round 100 from 88.10 to 91.35 (Appendix L.2).
Together these controls show contributions from both the sampling schedule and optimization, while the matched, tuned comparison establishes a likelihood-training advantage. The generator-transfer and prefix experiments identify what is learned: improved responses that help a reset student and an unchanged completer, beyond a decoding-temperature adjustment.
Dear Reviewer 9wuf,
We would like to gently follow up on our rebuttal, which includes validation and fixed-endpoint results, the controlled entropy-minimization comparison, and strong Gemma4 gains beyond Qwen. When you have a moment, could you please take a look and share your thoughts? Thank you again for your time and consideration.
Part 1/2
Thank you for your careful review and suggestions on model selection and objective comparisons.
How many checkpoints/configurations were considered, and how often were they evaluated?
The original nine-model comparison uses a common SePT recipe: AdamW with learning rate 10−7, one response per question and two updates per generated batch. Sampling starts at 0.6 for Qwen2.5-Math and DeepSeek-Math-Instruct, and 0.9 for the others, rising toward one over two epochs (Section 3.1).
For submitted Table 1, we use a 900-step checkpoint-selection window, saving and evaluating every 50 steps. We report the checkpoint with the highest benchmark AVG within that window.
Was validation separate, and do gains hold at a fixed training endpoint?
Original protocol. Submitted Table 1 uses the reported benchmarks for selection; its bootstrap intervals condition on those choices. This follows common benchmark-guided development practice in reasoning post-training (e.g., DAPO [1]). We now add both independent validation and a training duration fixed before evaluation.
New validation and endpoint results. We reserve 256 MATH training-split questions, stratified across seven subjects and difficulty levels 1–5 without model scores. Exact/near-match checks separate them from the six test benchmarks and remove matching validation/test questions from DSR training. Validation alone selects temperature, hyperparameters and the checkpoint; checkpoints are saved and evaluated every 25 rounds. These comparisons train for 300 rounds, fixed beforehand.
AVG averages Pass@1/8/32 across the paper's six benchmarks, with 32 responses per question. Validation selects Baseline temperature 0.4; trained models use τeval=1. The original learning rate is 10−7; tuning chooses 3×10−7 with the same schedule, averaged over two training runs. Substantial gains hold without checkpoint selection and with the original learning rate and sampling schedule (Section 3.7; Appendix L.1).
Beyond the principal Qwens. We also added text-only Gemma4 to the main-table comparison:
Gains are +11.40/+7.53 AVG points, with improvement in all three training runs (mean ± SD). Both methods use 32 responses and a 6,144-token cap; original-weight Baseline temperature is 1.5, SePT one (Table 1; Appendix L.3).
[1] Qiying Yu et al. DAPO: An Open-Source LLM Reinforcement Learning System at Scale. 2025