- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper proposes On-Policy Self-Training (OPST), which combines a self-generated curriculum with on-policy self-distillation. Starting from topic descriptors, the model generates questions and candidate solutions, checks answer consistency in a fresh context and under paraphrasing, and removes questions that the student already solves consistently. During training, the student sees only the question, while an EMA teacher additionally sees the generated solution and provides token-level distributional supervision along the student’s sampled trajectories. Experiments cover language tasks, mixed-domain training, and a visual extension that uses unlabeled images. The main contribution is removing the curated question set from this self-distillation pipeline.
Critical strengths
- The paper addresses a concrete remaining source of supervision. Removing reference answers or an external teacher still leaves many self-training methods dependent on externally supplied questions. OPST shows that questions can also be generated from coarse topic descriptions while retaining useful downstream gains. For example, Gemma 4 E2B without thinking improves from 37.5 to 47.7 on MMLU, and Gemma 4 E4B with thinking improves from 84.4 to 93.0 on GSM8K (Table 1). The Qwen3-4B-Base results in Appendix A.4 extend the evidence beyond instruction-tuned Gemma models. This makes the proposed combination worth examining even though the underlying on-policy distillation framework is established.
- The ablations substantiate several components of the complete pipeline. The privileged-information control includes both removal and a same-topic placebo. Their MMLU scores, 40.6 and 40.2, are below the full method’s 47.7, supporting a contribution from sample-specific information. The filtering ablation also shows a progression from 42.7 without filtering, to 45.4 with verification, to 47.7 with the consensus gate (Table 5). These results help establish that the pipeline is doing more than training on an arbitrary collection of self-generated text.
Critical weaknesses
. The incremental value over a self-generated curriculum plus simple self-distillation is not sufficiently isolated. The SFT baseline in Table 1 is relevant: the paper states that it uses the same self-generated data, and OPST performs better. However, this comparison changes several aspects of learning together, including the sampled trajectories, hard targets versus soft distributional supervision, and the use of privileged teacher context. The reporting also leaves unclear whether baseline curriculum regeneration, EMA settings, and the evaluated checkpoints are matched to OPST. Consequently, the statement that the SFT–OPST gap isolates using the generated answer as context rather than a target is too strong. A particularly informative control would retain the generated curriculum, filtering, privileged information, and EMA teacher, but let that teacher generate complete solutions for ordinary SFT. Distillation on teacher-generated trajectories would provide another control for the role of on-policy sampling. The existing SFT baseline directly imitates the generated privileged information, which can be a partial solution; it does not resolve whether a teacher allowed to use that information to generate a new solution would be sufficient. This distinction matters because it determines how much of the benefit requires the proposed training procedure beyond generating and selecting useful problems. The EMA ablation makes this concern more consequential: removing EMA reduces MMLU from 47.7 to 38.1, near the 37.5 starting model (Table 6). This demonstrates a strong dependence on smoothing within OPST; it does not establish that EMA alone explains the gain. Together with the privileged-information ablation, it motivates comparisons that control these components rather than attributing the full difference to a single design choice. 2. The training configurations and computational costs are insufficiently specified to assess the comparisons. The submitted text does not provide a complete account of essential settings such as the learning rate, optimizer, concrete optimization updates per round, or the number of consensus rollouts. It also does not report generation, verification, student rollout, and teacher-supervision costs alongside baseline budgets. Identical generated data does not imply matched computation. Without this information, it is difficult to reproduce the results or determine whether the added procedure offers a worthwhile advantage over simpler self-training. The rule for choosing the number of rounds also needs clarification: Table 6 shows the best reported MMLU result at seven rounds, but a selection or stopping rule usable without evaluation-benchmark feedback is not established.
Questions for authors
For the SFT baseline, are the filtering rules, curriculum regeneration schedule, number of rounds, and any EMA/evaluation settings matched to OPST? Please specify these choices so that the comparison with a simple iterative self-training pipeline is clear.
How does OPST compare with a teacher-generated-solution baseline that retains the same privileged information and EMA teacher but trains the student through ordinary SFT? Alternatively, can teacher-trajectory distillation under otherwise matched conditions isolate the contribution of on-policy sampling?
Can the authors provide complete training configurations and the total costs of generation, verification, rollouts, and optimization for each method? How are the training rounds or stopping point selected without using evaluation-benchmark performance?
5 comments
Response to Reviewer CPgT (2/2): Q3a costs, Q3b number of rounds
Q3a: Training configurations and costs
SFT, GRPO and OPST in Table 1 have the same costs for data generation, pseudo-verification and the consensus gate. The methods differ only in the training step.
| Stage | SFT | GRPO | OPST |
|---|---|---|---|
| Generation (per candidate) | 1 pass | 1 pass | 1 pass |
| Pseudo-verification (per candidate) | 3 passes | 3 passes | 3 passes |
| Consensus gate (per verified candidate) | 8 rollouts | 8 rollouts | 8 rollouts |
| Rollouts per training sample | 0 | 8 | 1 |
| Extra forward passes per training sample | 0 | 8 (reference) | 1 (privileged teacher) |
| Tokens trained on | z | 8 rollouts | 1 rollout |
Pseudo-verification solves the question, paraphrases it and solves the paraphrase. Samples with an incomplete rationale require one additional completion pass (Appendix A.7). SFT, GRPO and OPST use AdamW with a learning rate of 5 × 10⁻⁶, weight decay of 10⁻⁴, gradient-norm clipping at 0.1 and batch size 8, with 36 updates per round and G = 8 consensus rollouts.
Beyond the shared data pipeline, OPST requires one student rollout and one teacher forward pass per training sample. This is more than SFT, which requires neither, but only one eighth of the rollouts and forward passes GRPO needs. In measured terms, a full run on Gemma 4 E2B takes 5 hours on a TPU v5p for OPST, compared to 4 hours for SFT and 8 hours for GRPO. The supervised baselines skip data generation and filtering, but their cost excludes the human effort of curating questions and solutions. We add full training configurations and the measured costs to Appendix A.12.
Q3b: Selecting the number of rounds without evaluation-benchmark feedback
We selected all three hyperparameters (β, samples per round and number of rounds) on the MMLU validation split (1,531 questions), not on the test set. Appendix A.10 reports the validation accuracy of all settings. As a data-free method, OPST never uses benchmark data for training, so this split remains available for model selection. Its labels only select these three hyperparameters and never enter training. We tune them once and keep them fixed across models and benchmarks, so applying OPST to a new task requires no further validation data. The exact stopping point is also not critical: beyond round seven, MMLU drops by only 1.5 points, to 46.2, and stays there through round 20, well above the base model (37.5, Table 6c). A label-free stopping criterion remains future work.
We hope these controls resolve the reviewer’s questions, and we would be grateful if the reviewer would consider raising their score.
Correction to Table 10 (Appendix A.10)
Dear Reviewer CPgT,
We would like to share a brief update on Table 10 (Appendix A.10), which we added in the rebuttal revision and referred to in our answer to Q3b. While preparing the updated revision, we re-checked all entries against the evaluation logs and found that the validation accuracies for the default configuration, β = 0.20 and 0.01, 57, 171 and 570 samples per round, N = 3 rounds and the student checkpoint had been reported incorrectly. The revised PDF contains the corrected values.
The conclusions are unchanged. β = 0.05, 285 samples per round, N = 7 rounds and the EMA teacher remain the best settings on the validation split (46.7).
Best regards,
The Authors
Response to Reviewer CPgT (1/2): Q1 EMA-matched baselines, W1/Q2 controlled decomposition
We thank the reviewer for the careful review and the precise suggestions for controls, which we ran and report below.
Q1: Were the SFT baseline settings matched to OPST?
All methods in Table 1 use the same generated data, filtering rules, curriculum regeneration schedule and number of rounds. SFT and GRPO, however, are trained without EMA, as in their original formulation. To isolate the effect of EMA, we retrained SFT and GRPO in the same setting with the same EMA as OPST (β = 0.05, updated after every step) and evaluated the EMA weights. OPST already uses EMA, so its numbers are unchanged from Table 1. SFT trains with cross-entropy on the retained samples, using OPST’s answer formatting with 4× answer-token weighting. GRPO uses G = 8 rollouts with a binary reward for matching the pseudo-label.
Gemma 4 E2B, accuracy (%) of the EMA weights for all methods. OPST is unchanged from Table 1.
OPST outperforms SFT and GRPO on every benchmark and on average, with and without thinking, whether the baselines are evaluated with EMA (above) or without (Table 1). On E4B, against the better variant of each baseline with or without EMA, OPST leads SFT and GRPO by 1.3 and 4.9 points without thinking and by 3.8 and 4.3 points with thinking (response to Reviewer ATdm, Q1). We add both EMA tables to Appendix A.15.
W1/Q2: Isolating the source of OPST’s gains
We agree that the original SFT comparison changes several factors at once, and we revise the corresponding claim in Sec. 4.2. We ran both proposed controls in a controlled decomposition on Gemma 4 E2B and E4B, with and without thinking, across all four benchmarks. Each arm changes exactly one factor relative to the previous one:
All arms share the same generated curriculum, filters, privileged information, EMA teacher and seven rounds, use the same number of training samples and updates, and are evaluated with EMA weights after the final round. The differences between arms therefore cannot stem from EMA.
Macro-average accuracy (%), averaged as in Table 1, after round 7 with EMA weights, single seed.
Is a privileged teacher that generates solutions sufficient? No. OPST outperforms privileged-teacher SFT in all four regimes, by 2.4 points on average and by up to 3.5 points on E4B with thinking. Over the base model, OPST gains +5.7 points, compared to +3.3 for privileged-teacher SFT. We add the decomposition to Appendix A.14.