acceptodds
Under review as a conference paper at ICLR 2027

On-Policy Self-Training: Data-Free Self-Improvement in Large Language Models

Abstract

On-policy self-distillation enables post-training foundation models without relying on stronger external teachers. However, existing approaches rely on ground-truth question-answer pairs, human-curated problem sets, or other supervised signals. In this work, we reformulate on-policy self-distillation as a self-training framework, driven by self-generated question-answer pairs and privileged reasoning demonstrations. We push supervision to the limit: guided solely by coarse, high-level topic descriptions, the model synthesizes its own curriculum of queries and privileged reasoning traces without access to any human-curated prompts or answers. We find that self-distillation on fully synthetic privileged information enables a model to improve across multiple domains while retaining its capabilities on domains it is not self-trained on. We demonstrate the generality of this framework across text and multimodal domains, achieving consistent self-improvement on standard language benchmarks and Visual Question Answering. Our findings demonstrate that foundation models possess rich latent capabilities that can be amplified without external task supervision or curated problem distributions.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On-Policy Self-Distillation, Self-Training, Large Language Models, Vision Language Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_CPgTOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes On-Policy Self-Training (OPST), which combines a self-generated curriculum with on-policy self-distillation. Starting from topic descriptors, the model generates questions and candidate solutions, checks answer consistency in a fresh context and under paraphrasing, and removes questions that the student already solves consistently. During training, the student sees only the question, while an EMA teacher additionally sees the generated solution and provides token-level distributional supervision along the student’s sampled trajectories. Experiments cover language tasks, mixed-domain training, and a visual extension that uses unlabeled images. The main contribution is removing the curated question set from this self-distillation pipeline.

Critical strengths
  1. The paper addresses a concrete remaining source of supervision. Removing reference answers or an external teacher still leaves many self-training methods dependent on externally supplied questions. OPST shows that questions can also be generated from coarse topic descriptions while retaining useful downstream gains. For example, Gemma 4 E2B without thinking improves from 37.5 to 47.7 on MMLU, and Gemma 4 E4B with thinking improves from 84.4 to 93.0 on GSM8K (Table 1). The Qwen3-4B-Base results in Appendix A.4 extend the evidence beyond instruction-tuned Gemma models. This makes the proposed combination worth examining even though the underlying on-policy distillation framework is established.
  2. The ablations substantiate several components of the complete pipeline. The privileged-information control includes both removal and a same-topic placebo. Their MMLU scores, 40.6 and 40.2, are below the full method’s 47.7, supporting a contribution from sample-specific information. The filtering ablation also shows a progression from 42.7 without filtering, to 45.4 with verification, to 47.7 with the consensus gate (Table 5). These results help establish that the pipeline is doing more than training on an arbitrary collection of self-generated text.
Critical weaknesses

. The incremental value over a self-generated curriculum plus simple self-distillation is not sufficiently isolated. The SFT baseline in Table 1 is relevant: the paper states that it uses the same self-generated data, and OPST performs better. However, this comparison changes several aspects of learning together, including the sampled trajectories, hard targets versus soft distributional supervision, and the use of privileged teacher context. The reporting also leaves unclear whether baseline curriculum regeneration, EMA settings, and the evaluated checkpoints are matched to OPST. Consequently, the statement that the SFT–OPST gap isolates using the generated answer as context rather than a target is too strong. A particularly informative control would retain the generated curriculum, filtering, privileged information, and EMA teacher, but let that teacher generate complete solutions for ordinary SFT. Distillation on teacher-generated trajectories would provide another control for the role of on-policy sampling. The existing SFT baseline directly imitates the generated privileged information, which can be a partial solution; it does not resolve whether a teacher allowed to use that information to generate a new solution would be sufficient. This distinction matters because it determines how much of the benefit requires the proposed training procedure beyond generating and selecting useful problems. The EMA ablation makes this concern more consequential: removing EMA reduces MMLU from 47.7 to 38.1, near the 37.5 starting model (Table 6). This demonstrates a strong dependence on smoothing within OPST; it does not establish that EMA alone explains the gain. Together with the privileged-information ablation, it motivates comparisons that control these components rather than attributing the full difference to a single design choice. 2. The training configurations and computational costs are insufficiently specified to assess the comparisons. The submitted text does not provide a complete account of essential settings such as the learning rate, optimizer, concrete optimization updates per round, or the number of consensus rollouts. It also does not report generation, verification, student rollout, and teacher-supervision costs alongside baseline budgets. Identical generated data does not imply matched computation. Without this information, it is difficult to reproduce the results or determine whether the added procedure offers a worthwhile advantage over simpler self-training. The rule for choosing the number of rounds also needs clarification: Table 6 shows the best reported MMLU result at seven rounds, but a selection or stopping rule usable without evaluation-benchmark feedback is not established.

Questions for authors

For the SFT baseline, are the filtering rules, curriculum regeneration schedule, number of rounds, and any EMA/evaluation settings matched to OPST? Please specify these choices so that the comparison with a simple iterative self-training pipeline is clear.

How does OPST compare with a teacher-generated-solution baseline that retains the same privileged information and EMA teacher but trains the student through ordinary SFT? Alternatively, can teacher-trajectory distillation under otherwise matched conditions isolate the contribution of on-policy sampling?

Can the authors provide complete training configurations and the total costs of generation, verification, rollouts, and optimization for each method? How are the training rounds or stopping point selected without using evaluation-benchmark performance?

5 comments
AuthorsOpenReview ↗

Response to Reviewer CPgT (1/2): Q1 EMA-matched baselines, W1/Q2 controlled decomposition

We thank the reviewer for the careful review and the precise suggestions for controls, which we ran and report below.

Q1: Were the SFT baseline settings matched to OPST?

All methods in Table 1 use the same generated data, filtering rules, curriculum regeneration schedule and number of rounds. SFT and GRPO, however, are trained without EMA, as in their original formulation. To isolate the effect of EMA, we retrained SFT and GRPO in the same setting with the same EMA as OPST (β = 0.05, updated after every step) and evaluated the EMA weights. OPST already uses EMA, so its numbers are unchanged from Table 1. SFT trains with cross-entropy on the retained samples, using OPST’s answer formatting with 4× answer-token weighting. GRPO uses G = 8 rollouts with a binary reward for matching the pseudo-label.

ThinkingMethodMMLUGPQAGSM8KHellaSwagAvg.
×Base37.528.8–48.738.3
+ SFT41.929.2–53.141.4
+ GRPO36.930.8–48.438.7
+ OPST47.732.8–56.245.6
✓Base69.927.872.763.158.4
+ SFT70.931.874.963.760.3
+ GRPO70.431.374.864.360.2
+ OPST71.134.380.965.663.0

Gemma 4 E2B, accuracy (%) of the EMA weights for all methods. OPST is unchanged from Table 1.

OPST outperforms SFT and GRPO on every benchmark and on average, with and without thinking, whether the baselines are evaluated with EMA (above) or without (Table 1). On E4B, against the better variant of each baseline with or without EMA, OPST leads SFT and GRPO by 1.3 and 4.9 points without thinking and by 3.8 and 4.3 points with thinking (response to Reviewer ATdm, Q1). We add both EMA tables to Appendix A.15.

W1/Q2: Isolating the source of OPST’s gains

We agree that the original SFT comparison changes several factors at once, and we revise the corresponding claim in Sec. 4.2. We ran both proposed controls in a controlled decomposition on Gemma 4 E2B and E4B, with and without thinking, across all four benchmarks. Each arm changes exactly one factor relative to the previous one:

  1. SFT: the student imitates the generated answer (the EMA-trained SFT from Q1 and our response to Reviewer ATdm, Q1).
  2. Privileged-teacher SFT: the EMA teacher sees z and writes complete solutions, which the student imitates with SFT (the reviewer’s first control).
  3. Off-policy distillation: skewed-JSD distillation from the privileged teacher on the teacher’s own trajectories (the reviewer’s second control).
  4. OPST: the same distillation on the student’s own rollouts.

All arms share the same generated curriculum, filters, privileged information, EMA teacher and seven rounds, use the same number of training samples and updates, and are evaluated with EMA weights after the final round. The differences between arms therefore cannot stem from EMA.

RegimeBaseSFTPriv.-teacher SFTOff-policy distill.OPST
Gemma 4 E2B w/o think.38.341.442.843.245.6
Gemma 4 E2B w/ think.58.460.361.861.563.0
Gemma 4 E4B w/o think.46.148.950.250.952.2
Gemma 4 E4B w/ think.71.372.072.573.576.0
Average53.555.756.857.359.2

Macro-average accuracy (%), averaged as in Table 1, after round 7 with EMA weights, single seed.

  • Teacher conditioning adds +1.1 points, positive in all four regimes. Imitating the generated answer gains 2.2 points over the base model, and letting a teacher that sees z write a new solution improves on it in every regime, which confirms without confounds that the generated answer is more useful as context than as a target.
  • Soft targets add +0.5 points over imitating the teacher’s sampled solutions, which is within seed noise.
  • On-policy sampling contributes the most (+1.9 points), and the gain is positive in all four regimes.

Is a privileged teacher that generates solutions sufficient? No. OPST outperforms privileged-teacher SFT in all four regimes, by 2.4 points on average and by up to 3.5 points on E4B with thinking. Over the base model, OPST gains +5.7 points, compared to +3.3 for privileged-teacher SFT. We add the decomposition to Appendix A.14.

AuthorsOpenReview ↗

Response to Reviewer CPgT (2/2): Q3a costs, Q3b number of rounds

Q3a: Training configurations and costs

SFT, GRPO and OPST in Table 1 have the same costs for data generation, pseudo-verification and the consensus gate. The methods differ only in the training step.

StageSFTGRPOOPST
Generation (per candidate)1 pass1 pass1 pass
Pseudo-verification (per candidate)3 passes3 passes3 passes
Consensus gate (per verified candidate)8 rollouts8 rollouts8 rollouts
Rollouts per training sample081
Extra forward passes per training sample08 (reference)1 (privileged teacher)
Tokens trained onz8 rollouts1 rollout

Pseudo-verification solves the question, paraphrases it and solves the paraphrase. Samples with an incomplete rationale require one additional completion pass (Appendix A.7). SFT, GRPO and OPST use AdamW with a learning rate of 5 × 10⁻⁶, weight decay of 10⁻⁴, gradient-norm clipping at 0.1 and batch size 8, with 36 updates per round and G = 8 consensus rollouts.

Beyond the shared data pipeline, OPST requires one student rollout and one teacher forward pass per training sample. This is more than SFT, which requires neither, but only one eighth of the rollouts and forward passes GRPO needs. In measured terms, a full run on Gemma 4 E2B takes 5 hours on a TPU v5p for OPST, compared to 4 hours for SFT and 8 hours for GRPO. The supervised baselines skip data generation and filtering, but their cost excludes the human effort of curating questions and solutions. We add full training configurations and the measured costs to Appendix A.12.

Q3b: Selecting the number of rounds without evaluation-benchmark feedback

We selected all three hyperparameters (β, samples per round and number of rounds) on the MMLU validation split (1,531 questions), not on the test set. Appendix A.10 reports the validation accuracy of all settings. As a data-free method, OPST never uses benchmark data for training, so this split remains available for model selection. Its labels only select these three hyperparameters and never enter training. We tune them once and keep them fixed across models and benchmarks, so applying OPST to a new task requires no further validation data. The exact stopping point is also not critical: beyond round seven, MMLU drops by only 1.5 points, to 46.2, and stays there through round 20, well above the base model (37.5, Table 6c). A label-free stopping criterion remains future work.

We hope these controls resolve the reviewer’s questions, and we would be grateful if the reviewer would consider raising their score.

AuthorsOpenReview ↗

Correction to Table 10 (Appendix A.10)

Dear Reviewer CPgT,

We would like to share a brief update on Table 10 (Appendix A.10), which we added in the rebuttal revision and referred to in our answer to Q3b. While preparing the updated revision, we re-checked all entries against the evaluation logs and found that the validation accuracies for the default configuration, β = 0.20 and 0.01, 57, 171 and 570 samples per round, N = 3 rounds and the student checkpoint had been reported incorrectly. The revised PDF contains the corrected values.

The conclusions are unchanged. β = 0.05, 285 samples per round, N = 7 rounds and the EMA teacher remain the best settings on the validation split (46.7).

Best regards,

The Authors

Reviewer_rm4UOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes On-Policy Self-Training (OPST), a framework for data-free post-training of language and vision-language models. Starting only from coarse topic descriptors, OPST asks a model to generate its own questions, rationales, and candidate answers. After filtering, the generated samples are used for on-policy self-distillation: the student sees only the question, while an EMA teacher additionally receives the generated rationale and candidate answer as privileged information and provides token-level supervision along the student’s rollouts. Experiments show that OPST improves the performance over GRPO and SFT on the same self-generated data.

Critical strengths
  • The self-improvement of on-policy self-distillation from data side is an interesting research problem, which answer a question: whether a model can improve through post-training without pre-curated data, or an external teacher.
  • The paper further explores the potentials of self-training methods for eliciting and improving model reasoning capabilities. Unlike prior approaches that rely on curated questions, OPST shows that self-generated problems and privileged reasoning signals can provide useful supervision for eliciting reasoning capability of the model.
Critical weaknesses
  • The presentation of experimental results is overly table-centric, which makes the paper less readable. Beyond the method illustrations, the paper contains no figures to convey deeper analysis of the proposed method. For example, visualizations of the iterative self-training dynamics, generated-data quality, and other aspects of the experimental results would make the findings clearer and more informative.
  • The main limitation is that the quality-control mechanism remains entirely self-referential. Pseudo-verification checks whether the same model can reproduce an answer under a rephrasing, not whether the answer is actually correct. The paper does not quantify the factual correctness of its self-generated questions, rationales, or candidate answers.
  • The experimental comparisons and settings are limited. Self-training and self-rewarding of LLMs have already been studied recently, such as Intuitor, EMPO, Co-rewarding, R-Zero and etc.. Incorporating a broader range of recent self-training and self-rewarding methods would better enhance the empirical experiments and findings. Moreover, the evaluation covers only a small number of model settings: the main results are limited to Gemma models, and broader evaluation on widely used families such as Qwen and Llama, across different model scales 7B/8B, would provide stronger experiments of generality.

[1] Learning to Reason without External Rewards
[2] EMPO: Fully Unsupervised LLM Reasoning Incentivization
[3] Co-rewarding: Stable Self-supervised RL for Eliciting Reasoning in Large Language Models
[4] R-Zero: Self-Evolving Reasoning LLM from Zero Data

Questions for authors
  • Could the authors provide an external correctness audit of the self-generated questions, rationales, and candidate answers, both before and after pseudo-verification, and analyze how synthetic-data correctness relates to downstream performance?
  • Could the authors compare OPST with a broader set of recent self-training and self-rewarding methods with more model families and sizes?
7 comments
AuthorsOpenReview ↗

Response to Reviewer rm4U (1/2): W1 figures, W2/Q1 correctness audit

We thank the reviewer for the constructive review and the concrete suggestions, which we address below.

W1: Table-centric presentation

We have added two figures to the revised version to visualize the training dynamics of OPST and the effect of data quality. First, Figure 3 shows the self-training dynamics on Gemma 4 E2B: accuracy rises over the first seven rounds, peaks 10.2 points above the base model and then plateaus, well above SFT on the same synthetic data. Second, Figure 4 relates data quality to downstream accuracy on Gemma 4 E4B: as the share of faulty synthetic answers grows, OPST stays above the base model, whereas SFT falls below it.

W2/Q1: External correctness audit

We agree that pseudo-verification measures self-consistency rather than correctness. To test whether it still tracks correctness, we audited 1,792 synthetic candidates generated over all seven rounds by Gemma 4 E2B and E4B on the four benchmarks in both reasoning modes. Three external judges (Gemini 3.8 Flash, Gemini 3.1 Pro and Gemini 3.5 Flash-Lite) rated the question, the rationale and the answer of every sample at temperature 0. We calibrated the judges on 200 benchmark items with known labels, 50 each with a correct solution, a wrong answer, a flawed rationale and an ill-posed question. Their majority vote reaches 93.5% accuracy, accepts none of the 100 items with a wrong answer or a flawed rationale (95% upper bound 3.6%) and detects 96% of the ill-posed questions, so lenient judging is unlikely to inflate the rates below.

StageN (share of all)Well-posed questionCorrect answerValid and correctCorrect, given well-posedSound rationale
Before filtering1,792 (100%)43.437.533.476.932.4
After pseudo-verification254 (14.2%)73.274.469.795.270.1
After consensus gate (ours)114 (6.4%)75.475.471.194.271.9
Rejected by pseudo-verification1,495 (83.4%)39.332.328.271.626.9

Share of samples in % (three-judge majority vote). 43 candidates fail to parse and are discarded before pseudo-verification. Table 17 in Appendix A.13 gives every filter stage with confidence intervals.

  • Pseudo-verification roughly doubles data quality. Well-posed questions rise from 43.4% to 73.2%, sound rationales from 32.4% to 70.1% and correct answers from 37.5% to 74.4%. Rejected samples are valid and correct in only 28.2% of cases.
  • Remaining errors come mainly from ill-posed questions. Among well-posed retained questions, 94.2% to 95.2% have a correct answer, compared to 76.9% before filtering.
  • The consensus gate selects for difficulty, not quality. It leaves data quality unchanged (69.7% vs. 71.1% valid and correct), as intended.

Relation to downstream performance. We find that data quality and accuracy move together. Pseudo-verification, which doubles the share of valid and correct samples, raises MMLU from 42.7 to 45.4 (Table 5a). The consensus gate adds a further 2.3 points, to 47.7, without changing data quality, so this last step comes from selecting informative problems rather than cleaner ones. In the opposite direction, Table 5c disables both filters and replaces a growing share of candidate answers with wrong ones and a matching rationale. Since only about 37.5% of raw candidates are correct, the share of wrong answers rises from about 62% to 91%, and accuracy falls from 42.7 to 40.8, whereas SFT drops by 5.7 points. OPST therefore benefits from cleaner data but does not depend on it, because the teacher uses the generated answer as context rather than as a target.

AuthorsOpenReview ↗

Response to Reviewer rm4U (2/2): W3/Q2 self-rewarding baselines, Qwen3

W3/Q2: Comparison to more self-training and self-rewarding baselines

We extended the evaluation in two ways. First, we added Intuitor (Zhao et al., 2026b), EMPO (Zhang et al., 2025) and Co-Rewarding (Zhang et al., 2026) next to R-Zero, which is already in Table 2. Second, we evaluated all methods on the post-trained Qwen3-1.7B and Qwen3-4B, with and without thinking, and added Qwen3-8B as a larger model.

Self-rewarding baselines. Intuitor, EMPO and Co-Rewarding train on exactly the same self-generated questions as OPST. We use their official implementations and the same number of rollouts (G = 8). For Co-Rewarding, we use the data-side variant Co-Rewarding-I, which rewards each rollout for agreeing with the majority answer on a paraphrase of the same question, and vice versa. The paraphrases come from the unmodified base model, so no external model is involved. All three methods are answer-free by design, so they do not see the generated answers. R-Zero instead generates its own questions with a challenger model.

ModelThinkingBaseIntuitorEMPOCo-RewardingR-ZeroOPST
Gemma 4 E2B×38.338.238.438.341.845.6
✓58.461.161.361.461.663.0
Gemma 4 E4B×46.144.844.945.247.052.2
✓71.371.971.872.472.976.0
Qwen3-1.7B×45.345.545.345.747.150.1
✓66.067.067.067.266.867.5
Qwen3-4B×49.149.549.750.052.154.3
✓75.676.777.076.576.377.8

Macro-average accuracy (%), averaged as in Tables 1 and 2. Results for SFT and GRPO on the same data are in Table 1 and Appendix A.11.

  • OPST outperforms all self-rewarding methods. It is the best of these methods in all eight settings and leads Intuitor, EMPO and Co-Rewarding by 4.0, 3.9 and 3.7 points on average. With thinking on Qwen3, however, its lead over the strongest self-rewarding method (0.3 points over Co-Rewarding on Qwen3-1.7B and 0.8 over EMPO on Qwen3-4B) is within seed variance.
  • The gap is largest without thinking. In this setting, Intuitor, EMPO and Co-Rewarding change the base model by −1.3 to +0.9 points, while OPST gains 4.8 to 7.3 points. Their reward is a single scalar derived from the model’s own confidence or agreement, and it carries little signal for short answers. EMPO’s entropy gate also discards groups whose rollouts all agree, up to 45 of 48 groups on HellaSwag with Gemma 4. Co-Rewarding uses paraphrases, as our pseudo-verification does, but reduces them to a scalar agreement reward. OPST instead receives dense token-level targets on every retained problem, because the teacher sees the generated rationale and answer.

Xuandong Zhao et al. Learning to Reason without External Rewards. International Conference on Learning Representations (ICLR 2026).

Qingyang Zhang et al. Right Question is Already Half the Answer: Fully Unsupervised LLM Reasoning Incentivization. Advances in Neural Information Processing Systems 38 (NeurIPS 2025).

Zizhuo Zhang et al. Co-rewarding: Stable Self-supervised RL for Eliciting Reasoning in Large Language Models. International Conference on Learning Representations (ICLR 2026).

Model families and sizes. On Qwen3, OPST also matches or exceeds all curated-data baselines in three of four settings, and the spurious-reward controls of Shao et al. (2026) do not reproduce its gains (response to Reviewer ATdm, Q7). On the larger Qwen3-8B, OPST gains as much as on the smaller models:

Qwen3-8B, w/o thinkingMMLUGPQAHellaSwagAvg. (Δ)
Base70.832.160.454.4
SFT72.1 (+1.3)38.1 (+6.0)68.4 (+8.0)59.5 (+5.1)
GRPO71.7 (+0.9)37.1 (+5.0)61.8 (+1.4)56.9 (+2.5)
R-Zero71.5 (+0.7)35.6 (+3.5)62.0 (+1.6)56.4 (+2.0)
u-OPSD70.5 (−0.3)37.1 (+5.0)65.9 (+5.5)57.8 (+3.4)
OPST71.5 (+0.7)39.6 (+7.5)68.2 (+7.8)59.8 (+5.4)

Qwen3-8B without thinking, accuracy (%) with the change over the base model.

OPST achieves the best average, slightly ahead of SFT (59.8 vs. 59.5) and ahead of GRPO, R-Zero and u-OPSD, and gives the largest GPQA gain of all methods (+7.5). All tables are in Appendix A.11.

We hope these additions address the reviewer’s concerns, and we would be grateful if the reviewer would reconsider their score.

AuthorsOpenReview ↗

Brief follow-up before the discussion deadline

Dear Reviewer rm4U,

With the discussion period closing tomorrow, we wanted to kindly check in to see if our rebuttal and revised paper addressed your concerns. Following your suggestions, we added new figures on training dynamics and data quality, ran an external correctness audit across 1,792 generated samples, and added comparisons to Intuitor, EMPO, and Co-Rewarding as well as Qwen3 models up to 8B.

We are happy to clarify anything else before the deadline, and if your concerns have been resolved, we would really appreciate it if you considered updating your score.

Best regards,

The Authors

Reviewer_ATdmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes On-Policy Self-Training (OPST). It is a version of on-policy self-distillation (OPSD) that removes both the curated problem set and the reference answers. Training is seeded only by coarse topic descriptors such as "college_mathematics". From each topic, the model generates a question x and privileged information z, which consists of a rationale and a candidate answer. Samples are kept only if they pass two filters:

  • Pseudo-verification: the generator must reproduce the candidate answer when it solves the question in a fresh context and when it solves a paraphrase of it.
  • Consensus gate: samples are discarded if every student rollout already agrees with the candidate answer.

The student sees only x. It is trained to match, token by token and with a skewed JSD, a teacher that also sees z. The teacher is an EMA of the student. Generation, filtering and training repeat over several rounds (7 by default), and the EMA teacher is the model that is evaluated. A visual variant uses unlabeled images, with self-written captions and spatial relations derived from self-detected boxes as z.

Claimed contributions:

  1. A data-free post-training setting in which the model creates its own problems and supervision.
  2. The OPST pipeline (topic-seeded generation, the two filters, OPSD with an EMA teacher, iterative rounds).
  3. Evidence that using a generated answer as teacher context beats using it as an SFT target or as a GRPO reward.
  4. An extension to vision-language models.

Validation is done on Gemma 4 E2B/E4B, with and without thinking, on MMLU, GPQA-Diamond, GSM8K and HellaSwag. OOD benchmarks (ARC-C, MATH-500, Winogrande) are evaluated for E2B multi-domain mixtures only. Visual OPST is tested on VQAv2, POPE and CountBench. Qwen3-4B-Base appears in the appendix. Headline results:

  • +10.2 MMLU on E2B without thinking.
  • OPST beats SFT and GRPO on the same generated data.
  • OPST beats u-OPSD and R-Zero.
  • With thinking, OPST matches supervised OPSD.
  • Visual OPST gives +6.9 on VQAv2.
Critical strengths

S1. A meaningful step toward closed-loop, data-free post-training. The paper correctly observes that each post-training paradigm "relocates rather than removes" supervision (Sec. 1). OPST goes further than prior label-free self-training by removing the curated problem set as well as the answers. The generator, the verifier and the distillation teacher are the same EMA model and improve together across rounds (Sec. 3.6). The consensus gate keeps the curriculum tied to the current student (Sec. 3.4). This matters to my assessment because it is a concrete, working move toward automated model improvement without human-curated data, a direction of broad interest.

I would frame it as eliciting and amplifying latent capability rather than open-ended recursive self-improvement, for two reasons:

  • The authors state that OPST "can only amplify existing knowledge" (Sec. 6).
  • Gains stop growing after 7 rounds (Table 6c) and after 285 samples per round (Table 6b).

S2. Consistent gains, with ablations that tie them to the core design choice. On identical self-generated data, OPST beats SFT by 5.6 / 1.3 average points (E2B / E4B) without thinking and by 2.7 / 3.8 with thinking, and it also beats GRPO (Table 1). Among methods without reference solutions, it has the best average in all four settings (Table 2). The ablations support the mechanism:

  • Removing z or replacing it with a placebo from another problem cuts the MMLU gain from +10.2 to +3.1 or +2.7 (Table 5b).
  • Each filter adds accuracy (Table 5a).
  • With 75% of answers corrupted, OPST loses far less than SFT: −1.9 vs −5.7 (Table 5c).

There are also multi-domain, visual, EMA, data-size, round-count and seed experiments (Tables 3, 4, 6, 7). This matters because it supports the paper's central claim that a noisy generated answer is more useful as teacher context than as a training target, and that claim could apply beyond this pipeline.

Critical weaknesses

W1. The quality and scalability of the self-generated training data are neither controlled nor measured, and what makes a good z is undefined. The authors acknowledge that generated samples "can contain ill-posed questions or incorrect candidate answers" (Sec. 3.4). They also acknowledge that pseudo-verification "measures self-consistency rather than correctness, [so] consistent errors still enter training" (Sec. 6). Both filters are therefore basic self-consistency checks run by near-copies of the same model, not indicators of question quality. Specific gaps:

  • No data audit. The paper reports no correctness rate for retained samples, no filter pass rates, no difficulty or diversity statistics, no examples, and no decontamination against test sets.
  • z is not optimized. It is sampled once from the generator (Eq. 4) and never selected for helpfulness or correctness. A.1 gives only an informal argument for why it helps.
  • The most useful part of z is never checked. Both filters check only the answer, yet the rationale carries more signal than the answer (rationale only +8.3 vs answer only +7.1, Table 5b). When the rationale is "incomplete", it is regenerated conditioned on the candidate answer (App. A.7), which is post-hoc rationalization.
  • The gate keeps likely-wrong samples. It discards only samples where all rollouts agree (c = 1). Samples where no rollout agrees (c = 0) are kept, and these are the most likely to carry a wrong answer. Unlike DAPO's dynamic sampling (Yu et al., 2025), it does not drop both extremes. The claims that retained samples "lie at the student's capability frontier" and that difficulty "evolves with the model" have no supporting statistics.
  • The robustness test does not cover realistic errors. Table 5c randomly replaces answers, apparently leaving the rationale intact. At 75% corruption, OPST (40.8) falls to its no-privileged-information level (40.6), which suggests the teacher stops using z rather than correcting it. Plausible, self-consistent errors are not tested.
  • Scalability to harder tasks is not shown. Models are ≤4B (acknowledged). The tasks are mostly 4-way multiple choice and grade-school math, while a 2B model is prompted for "graduate-level" questions. MATH-500 regresses by 8.5 and 4.9 points in two of the mixtures (Table 3). Gains plateau in both rounds and data size (Table 6b–c).

This weakness is consequential because the paper's main claim is that the model can supervise itself. Without evidence on data quality, it is unclear whether the gains come from learning from the self-generated problems, or mainly from EMA self-distillation and format or calibration effects. The no-z variant already gives +3.1 on MMLU (Table 5b).

W2. The empirical comparison does not establish the claimed advantage or its generality: the existing-data OPSD comparison is incomplete, the comparisons are not shown to be fair, and only one model family has baselines.

  • Comparison to OPSD on existing data. Table 2 includes supervised OPSD and u-OPSD trained on curated data. However:

    • The source, split and size of that curated data are never specified. GPQA has no standard training split, and Diamond is a subset of the main set.
    • The amount of curated data is not matched to OPST's roughly 2k generated samples (285 per round × 7 rounds).
    • Without thinking, supervised OPSD still beats OPST: 46.6 vs 45.6 at E2B, 54.3 vs 52.2 at E4B.
    • Every evaluated domain has curated data available, so the experiments never test the setting that motivates the method, where such data is missing.
    • Existing-data OPSD is absent from the multi-domain/OOD (Table 3), visual (Table 4) and Qwen (Table 8) experiments.

    Two additions would show when data-free self-training is necessary or complementary: an accuracy-vs-number-of-curated-samples curve for supervised OPSD, and OPST combined with curated data.

  • Fairness and reporting.

    • Only OPST is evaluated as an EMA model (Sec. 4.1). EMA alone adds +1.8 (Table 6d), more than the OPST–SFT gap at E4B without thinking (+1.3).
    • Every default hyperparameter (β, samples per round, number of rounds) is the best value on the MMLU test set (Table 6a–c), and no validation split is mentioned.
    • u-OPSD and R-Zero are re-implemented with no reported hyperparameters, and no method's training budget or compute is given.
    • Seeds are reported only for OPST (Table 7). Several margins are within about 1 std: vs u-OPSD 45.6 vs 44.6; vs SFT at E4B 52.2 vs 50.9; vs supervised OPSD with thinking +0.2 / +0.7.
    • The largest gain (+10.2 MMLU, no thinking) is measured with log-likelihood scoring, where the base model scores 37.5 against 69.9 with thinking. With thinking, the MMLU gain is only +1.2.
  • Generality across models. All baselines are on Gemma 4. The Qwen3-4B-Base result (Table 8) reports only Base vs OPST. Qwen models are known to improve even under spurious training signals (Shao et al., 2025), so a gain without baselines says little about OPST specifically. The visual results (Table 4) also have no baselines. No model larger than the 4B class or from a third family is tested.

This weakness is consequential because the paper's case for acceptance rests on OPST being competitive with methods that use curated data, and better than alternatives on the same data. The current setup cannot separate that from evaluation-protocol choices, EMA averaging and test-set tuning, or show that it holds beyond one model family.

Questions for authors

Clarifications (answers could change my assessment):

  1. What curated data (source, split, number of samples) did supervised SFT, supervised OPSD and u-OPSD use, especially for GPQA? Was it matched in size to OPST's generated data? Were these baselines also evaluated with EMA weights?
  2. How were β, samples per round and number of rounds selected? Was any held-out validation data used, or only the reported test benchmarks?
  3. What fraction of retained samples have correct answers, as judged by a stronger model or humans on a subset, and how does this change across rounds? What are the per-filter pass rates? What fraction of retained samples have c = 0?
  4. In Table 5c, is the rationale also corrupted when the answer is replaced?
  5. Were generated questions checked for overlap with the evaluation sets? The topics are the benchmark subject labels and the templates copy the benchmark formats.
  6. Can you report generative (not log-likelihood) accuracy without thinking, to rule out option-likelihood recalibration as the source of the +10.2 MMLU gain?

Suggestions (experiments that would strengthen the paper):

  1. Add SFT, GRPO, u-OPSD and supervised-OPSD baselines on Qwen3-4B and on at least one larger or different model family, and add baselines for Visual OPST.

References

  • Snell, C., Klein, D., & Zhong, R. (2022). Learning by Distilling Context. arXiv:2209.15189.
  • Yu, Q., et al. (2025). DAPO: An Open-Source LLM Reinforcement Learning System at Scale. arXiv:2503.14476.
  • Shao, R., et al. (2025). Spurious Rewards: Rethinking Training Signals in RLVR. arXiv:2506.10947.
11 comments
AuthorsOpenReview ↗

Response to Reviewer ATdm (1/9): Framing, Q1 curated data

We thank the reviewer for the thorough review and the clear list of clarification questions. We agree with the suggested framing. OPST elicits and amplifies latent capabilities rather than performing open-ended recursive self-improvement, consistent with our limitations section. Below, we first answer the clarification questions and then address the remaining points.

Q1: Curated data and EMA for baselines

Curated data. We should have specified these details in the paper, and we add them to Appendix A.9. We match the data budget of OPST on every benchmark: all curated baselines train on 1,995 curated samples, the same number of samples OPST trains on (285 samples per round retained after filtering, over 7 rounds, Table 6b–c). The curated data comes from the following sources:

  • GPQA: GPQA has no training split, so we sample 1,995 questions from SuperGPQA (Du et al., 2025) that cover the topic labels OPST uses for GPQA. SuperGPQA questions have up to ten options, so we keep the correct option and three randomly sampled distractors, shuffle them and relabel the answer to match the four-option format of GPQA. Sup-SFT and Sup-OPSD use the gold answer. u-OPSD uses the questions without labels.
  • MMLU: auxiliary_train, the training split released with MMLU, which draws on ARC, MCTest, OBQA and RACE. The base model of each run (Gemma 4 E2B or E4B) assigns each question to its closest topic label among those OPST uses for MMLU, and we sample 1,995 questions evenly across these topics. Sup-SFT and Sup-OPSD use the gold answer. u-OPSD uses the questions without labels.
  • GSM8K: official train split, disjoint from the test split. Sup-SFT and Sup-OPSD use the human-written step-by-step solutions and gold answers. u-OPSD uses only the problem statements.
  • HellaSwag: official train split, disjoint from the validation split we evaluate on. Sup-SFT and Sup-OPSD use the gold ending. u-OPSD uses the context and the four endings without labels.

On every benchmark, Sup-SFT and Sup-OPSD receive the same supervision, so their comparison isolates how the label is used. Sup-OPSD needs no reference rationales, since the privileged teacher produces the reasoning during training.

AuthorsOpenReview ↗

Response to Reviewer ATdm (2/9): Q1 EMA

Q1 (continued): EMA

EMA. SFT and GRPO in Tables 1 and 2 are trained without EMA, as in their original formulation. Sup-OPSD likewise follows the original OPSD (Zhao et al., 2026a), which uses the frozen initial model as teacher and no EMA. Sup-SFT, u-OPSD and R-Zero were also originally evaluated without EMA weights. To isolate the effect of EMA, we retrained all baselines with the same EMA as OPST (β = 0.05) and evaluated the EMA weights. OPST already uses EMA, so its numbers are unchanged from Table 1. EMA changes the baselines by −2.0 to +1.4 points with no consistent direction, so we compare OPST against the better of the two variants of each baseline.

On E2B, OPST remains ahead of SFT and GRPO on average and on every benchmark, with 45.6 vs. 41.4 (SFT) and 39.3 (GRPO) without thinking and 63.0 vs. 60.3 and 60.2 with thinking. On E4B, all methods compare as follows:

Gemma 4 E4Bw/o thinking, no EMAw/o thinking, EMAw/ thinking, no EMAw/ thinking, EMA
SFT50.948.972.272.0
GRPO47.346.671.371.7
R-Zero47.046.572.971.1
u-OPSD49.851.073.274.1
OPST–52.2–76.0
With reference answers
Sup-SFT52.352.374.273.8
Sup-OPSD54.352.675.374.0

Average accuracy (%), following Table 1 (three benchmarks without thinking, four with thinking). No-EMA values are from Tables 1 and 2. OPST is always evaluated with EMA.

EMA alone adds 1.8 points (Table 6d), more than the original gap to SFT without thinking (1.3). Applying the same EMA to SFT does not close this gap: its EMA weights score 48.9, below its original 50.9, so OPST remains 1.3 points ahead of SFT’s better variant. Without thinking, OPST leads all methods without reference answers, including u-OPSD, which trains on curated questions (52.2 vs. 51.0). It is on par with Sup-SFT (52.2 vs. 52.3) and trails only Sup-OPSD, by 2.1 points. With thinking, OPST leads every method, including both supervised baselines: by 0.7 points over Sup-OPSD and by 1.8 to 4.3 points over all others.

EMA benefits OPST more than the baselines because it plays a different role. In OPST, the EMA model is an essential part of the method and provides dense supervision to the student. This is the same target-network effect as in BYOL (Grill et al., 2020), DINO (Caron et al., 2021) and Mean Teacher (Tarvainen and Valpola, 2017): a slowly moving teacher provides stable targets and stops the loop from amplifying its own errors. For the baselines, EMA only averages the weights over training and cannot change the training signal. Even the OPST student, evaluated without weight averaging, outperforms SFT on E2B MMLU (45.9 vs. 41.9 with EMA and 38.3 without, Tables 6d and 1), so the gain does not stem from EMA averaging alone. Our response to Reviewer CPgT (Q1) gives the full E2B table and setting.

Siyan Zhao et al. Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models. Proceedings of the 43rd International Conference on Machine Learning (ICML 2026), PMLR.

Jean-Bastien Grill et al. Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning. Advances in Neural Information Processing Systems 33 (NeurIPS 2020).

Mathilde Caron et al. Emerging Properties in Self-Supervised Vision Transformers. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV 2021).

Antti Tarvainen, Harri Valpola. Mean Teachers are Better Role Models: Weight-Averaged Consistency Targets Improve Semi-Supervised Deep Learning Results. Advances in Neural Information Processing Systems 30 (NeurIPS 2017).

AuthorsOpenReview ↗

Response to Reviewer ATdm (3/9): Q2 hyperparameters, Q3 data audit

Q2: Selection of β, samples per round and number of rounds

We selected all three hyperparameters on the MMLU validation split (1,531 questions), not on the test set. As a data-free method, OPST never trains on any benchmark data, so this split remains available for model selection. We select the values once and keep them fixed for all benchmarks and models, including Qwen3, so the GPQA, GSM8K and HellaSwag results involve no tuning on their own data. Table 6 reports test accuracy only to show that the selected configuration transfers well. We swept β, the number of samples retained per round after filtering (Table 6b) and the number of rounds (Table 6c) on E2B. We report the validation accuracy of all settings in Table 10 (Appendix A.10).

Q3: Correctness of retained samples, pass rates and c = 0

We agree that pseudo-verification measures self-consistency rather than correctness. To test whether it still tracks correctness, we audited 1,792 synthetic candidates generated over all seven rounds by Gemma 4 E2B and E4B on the four benchmarks in both reasoning modes. In training, OPST retains 285 samples per round after filtering. Three external judges (Gemini 3.8 Flash, Gemini 3.1 Pro and Gemini 3.5 Flash-Lite) rated every sample at temperature 0 on three criteria: a well-posed question, a sound rationale and a correct answer. We first calibrated the judges on 200 benchmark items with known labels, 50 each with a correct solution, a wrong answer, a flawed rationale and an ill-posed question. Their majority vote reaches 93.5% accuracy [89.2, 96.2] and accepts none of the 100 items with a wrong answer or a flawed rationale (95% upper bound 3.6%), so lenient judging is unlikely to inflate the rates below. Inter-judge agreement is κ = 0.87 (Cohen) and 0.80 (Fleiss).

StageNPass rateWell-posedCorrect answerValid and correct [95% CI]Correct, given well-posedSound rationale
All candidates1,792100.043.437.533.4 [31.2, 35.6]76.932.4
Parsed1,74997.6 (97.6)44.338.434.2 [32.0, 36.4]77.333.2
+ Fresh solve41523.2 (23.7)65.365.560.0 [55.2, 64.6]91.958.3
+ Paraphrase solve25414.2 (61.2)73.274.469.7 [63.8, 75.0]95.270.1
+ One-sided gate (c < 1)20011.2 (78.7)73.074.569.0 [62.3, 75.0]94.570.0
+ Two-sided gate (1/8 ≤ c < 1)1659.2 (65.0)73.374.569.1 [61.7, 75.6]94.269.7
+ Two-sided gate (2/8 ≤ c < 1)1407.8 (55.1)75.075.770.7 [62.7, 77.6]94.371.4
+ Two-sided gate (3/8 ≤ c < 1), ours1146.4 (44.9)75.475.471.1 [62.1, 78.6]94.271.9
Rejected by pseudo-verification1,495–39.332.328.2 [25.9, 30.5]71.626.9

Share of samples in % (three-judge majority vote). The pass rate is relative to all candidates, with the share of the previous stage in parentheses. The gate rows are relative to the pseudo-verified samples.

  • Pseudo-verification raises quality. It more than doubles the share of samples that are both well-posed and correct, from 34.2% to 69.7%, and the share of sound rationales, from 33.2% to 70.1%. Among well-posed retained questions, 94.2% to 95.2% have a correct answer, compared to 77.3% before filtering. Rejected samples are valid and correct in only 28.2% of cases.
  • The consensus gate selects for difficulty, not quality. It leaves data quality unchanged (69.0% to 71.1% valid and correct), as intended.
  • Pass rates per filter. Parsing passes 97.6% of candidates, the fresh solve 23.7% of those and the paraphrase solve 61.2% of those. Our consensus gate (c ≥ 3/8) keeps 44.9% of the pseudo-verified samples. It removes 21.3% that the student already solves unanimously and 33.9% with low agreement.
  • c = 0. Under a strictly one-sided gate (c < 1), as Sec. 3.4 writes it, 17.5% of retained samples have c = 0 (35 of 200). All our experiments require c ≥ 3/8, so these samples are discarded, and we correct the definition of the consensus gate in Sec. 3.4 accordingly (see W1). Dropping them leaves data quality unchanged (69.1% vs. 69.0% valid and correct).
  • Stable across rounds. The pass rate of pseudo-verification rises slightly from 13.3% in round 1 to 15.6% in round 7, and the quality of retained samples shows no systematic trend. Diversity remains stable (distinct-4 between 0.81 and 0.82, 1 − Jaccard₄ ≈ 0.96), so we observe no mode collapse.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

We thank all reviewers. They found OPST "a meaningful step toward closed-loop, data-free post-training" (ATdm) that removes "a concrete remaining source of supervision" (CPgT). Our responses to each reviewer cover all concerns, and the revised paper includes the new experiments. The main points we address are:

1. External audit of the generated data (ATdm, rm4U)

  • Three external judges, calibrated on labeled benchmark items, rated 1,792 generated samples. Pseudo-verification doubles the share of valid and correct samples, from 34% to 70%, and does the same for sound rationales, even though it only checks the answer (ATdm Q3, rm4U W2/Q1).
  • The consensus gate selects for difficulty, not quality. It leaves data quality unchanged and discards questions that the student already solves or barely agrees on. Quality and diversity stay stable over all seven rounds, with no sign of mode collapse (ATdm Q3, rm4U W2/Q1).
  • As the share of faulty answers grows, OPST stays above the base model on Gemma 4 E4B, while SFT falls 26.5 points below it. OPST uses the generated answer as context rather than as a target, which makes it robust to these errors (rm4U W1, W2/Q1).

2. More model families and scales (ATdm, rm4U)

  • On Qwen3-1.7B and Qwen3-4B, OPST uses no curated data and still matches or beats all curated-data baselines, including supervised OPSD, in 3 of 4 settings (ATdm Q7).
  • On the larger Qwen3-8B, OPST gains +5.4 points, as much as on the smaller models, and achieves the best average. Spurious rewards gain at most +2.0, and OPST with permuted z falls below the base model (ATdm Q7, rm4U W3/Q2).
  • Visual OPST outperforms the new Visual SFT and Visual GRPO baselines, trained on the same unlabeled images, on VQAv2 and POPE for both Gemma 4 models, and improves VQAv2 by up to 6.9 points (ATdm Q7).

3. Self-training and self-rewarding baselines (rm4U)

  • We add Intuitor, EMPO and Co-Rewarding as additional baselines, trained on the same self-generated questions as OPST (rm4U W3/Q2).
  • OPST outperforms all three in all 8 settings, by 3.7 to 4.0 points on average. The gap is largest without thinking, where their scalar rewards change the base model by only −1.3 to +0.9 points, while OPST’s dense token-level targets gain 4.8 to 7.3 points (rm4U W3/Q2).

4. Controlled decomposition (CPgT, ATdm)

  • Following the controls CPgT proposed, each experiment changes one factor at a time, with the same EMA teacher across all of them. Teacher conditioning (+1.1) and on-policy sampling (+1.9) both contribute in every regime (CPgT W1/Q2).
  • SFT on solutions written by the privileged teacher is not enough. OPST leads it by 2.4 points on average and by up to 3.5 points on Gemma 4 E4B with thinking (CPgT W1/Q2).
  • With EMA-matched baselines, OPST still beats SFT and GRPO in every Gemma 4 setting, so the gain does not come from EMA alone (CPgT Q1, ATdm Q1).

5. Figures and implementation details (rm4U, CPgT, ATdm)

  • New figures show the self-training dynamics and how data quality relates to accuracy. Accuracy rises over seven rounds, peaks 10.2 points above the base model and then plateaus (rm4U W1).
  • We report full training configurations and measured costs. A full run on Gemma 4 E2B takes 5 hours for OPST, compared to 4 hours for SFT and 8 hours for GRPO (CPgT Q3a, ATdm W2).
  • All curated-data baselines train on 1,995 samples from documented sources, matching the data budget of OPST (ATdm Q1). All E2B baselines now report three seeds (ATdm W2).
  • Hyperparameters are selected once on the MMLU validation split, not the test set, and kept fixed across all models and benchmarks (ATdm Q2, CPgT Q3b).

Detailed results and answers to all specific questions are in our responses to each reviewer. We hope this resolves the shared concerns and would be grateful if the reviewers considered it in their final assessment.

Area_Chair_CUpZOpenReview ↗

Urgent Reminder: Reviewer Discussion Deadline

Dear Reviewers,

This is a friendly but urgent reminder that the discussion period will end on November 18, 2026. Please carefully review the authors' rebuttals and actively participate in the ongoing discussions. If you have not yet responded to the authors or initiated a discussion, please do so as soon as possible. In particular, reviewers are expected to address the authors' clarifications and responses, provide constructive follow-up feedback, and update their assessments and scores when appropriate. Please ensure that your final evaluations reflect your independent judgment and take into account the information provided during the rebuttal and discussion process. Active and timely participation is essential to ensuring a fair, thorough, and constructive review process.

Thank you for your time, effort, and cooperation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.